A method, system and medium for detecting unsafe behavior of personnel at a tunneling working face

Abstract

The present application discloses a method, system and medium for detecting unsafe behaviors of personnel in a tunneling face, and relates to the field of image data processing, including: installing an infrared thermal imager and a visible light camera in a preset area of ​​the tunneling face, collecting thermal imaging images and binocular images on both sides of the tunneling machine, using an image analysis algorithm to delineate a belt monitoring area on the thermal imaging image and set temperature and area thresholds, and delineating a virtual restricted area and a three-dimensional virtual tripwire containing height information on the binocular image. In the production process, the presence of a living heat source in the belt monitoring area is determined by analyzing the thermal imaging image, and the presence of personnel in violation of regulations and dangerous behaviors is determined by analyzing the binocular image. Finally, a comprehensive determination is made based on the analysis results of the thermal imaging and binocular images to determine whether the personnel have unsafe behaviors, etc.; in view of the low efficiency of supervision of coal mine tunneling faces in the prior art, the present application improves the supervision efficiency.

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a method, system and medium for detecting unsafe behaviors of personnel at an excavation working face. Background Art

[0002] Coal mining is a highly dangerous production activity, among which the excavation face is one of the most accident-prone areas underground. The excavation face has a harsh environment, narrow space, and intensive operation of various large-scale machinery and equipment. In addition, the lighting conditions are poor and the dust concentration is high, which can easily lead to personnel misoperation or illegal operation, causing mechanical injuries, roof collapse, gas explosions and other accidents, which seriously threaten the life safety of the operators. Therefore, real-time and effective supervision of the behavior of personnel in the excavation face and timely discovery and elimination of unsafe factors are crucial to curbing accidents and ensuring safe production in coal mines.

[0003] Traditional safety supervision of excavation working faces mainly relies on manual inspections and video surveillance. However, due to the complex and changeable environment of the excavation working face, it is difficult for manual inspections to achieve full coverage and no blind spots, and the inspection frequency and route are fixed, which is easy to be circumvented by violators. Although the video surveillance system can achieve continuous and uninterrupted monitoring, it is difficult to guarantee image quality due to factors such as insufficient light and dust obstruction. In addition, ordinary visible light cameras can only obtain planar information, making it difficult to accurately judge the spatial position relationship between personnel and dangerous areas or equipment. In addition, traditional video surveillance lacks intelligent analysis capabilities and must be monitored by a dedicated person. There are problems such as low supervision efficiency, lack of real-time performance, and easy fatigue and missed judgments, which makes it difficult to meet the growing safety management needs of coal mines. Summary of the invention

[0004] In view of the problem of low efficiency in monitoring coal mine excavation working faces in the prior art, the present application provides a method, system and medium for detecting unsafe behaviors of personnel in excavation working faces, by installing infrared thermal imagers and visible light cameras in preset areas of the excavation working faces, collecting thermal imaging images and binocular images on both sides of the excavator, using image analysis algorithms to delineate belt monitoring areas on thermal imaging images and set temperature and area thresholds, and delineating virtual restricted areas and three-dimensional virtual tripwires containing height information on binocular images. During the production process, the presence of a living heat source in the belt monitoring area is determined by analyzing thermal imaging images, and the presence of personnel in violation of regulations and dangerous behaviors is determined by analyzing binocular images. Finally, the analysis results of thermal imaging and binocular images are used to comprehensively determine whether personnel have unsafe behaviors, etc., thereby improving the efficiency of monitoring.

[0005] The purpose of this application is achieved through the following technical solutions.

[0006] The present application provides a method for detecting unsafe behaviors of personnel at a tunneling working face, comprising: installing an infrared thermal imager and a visible light camera in a preset area of ​​the tunneling working face: the infrared thermal imager is used to collect thermal imaging images on both sides of the tunneling machine to obtain infrared radiation signals of the target area; the visible light camera is used to collect binocular images on both sides of the tunneling machine to obtain visible light color and texture information of the target area; an image analysis algorithm is used to delineate a monitoring area of ​​a preset range of a belt on the thermal imaging image, and a temperature alarm threshold and a target area threshold are set; an image analysis algorithm is used to delineate a virtual restricted area and a virtual trip wire on the binocular image, and a violation database is established: a virtual restricted area is set, and two trip wires are set on the boundary of the virtual restricted area; A virtual tripwire is used as a dangerous area where users are prohibited from entering. A binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space, and a three-dimensional virtual tripwire containing the height information is generated. During the production process, thermal imaging images are analyzed to determine whether there are targets that meet the living judgment standards in the belt surface monitoring area. During the production process, binocular images are analyzed to determine whether the user has violated regulations and engaged in dangerous behaviors. Based on the analysis results of the thermal imaging images and the binocular images, it is determined whether the user has engaged in unsafe behaviors: when thermal imaging detects the presence of a living heat source in the belt monitoring area, and the binocular image analysis finds that a person is in a dangerous area or has made an illegal move, it is considered an unsafe behavior.

[0007] Furthermore, an image analysis algorithm is used to delineate a monitoring area of ​​a preset belt range on the thermal imaging image, including: preprocessing the thermal imaging image to extract the belt edge image; using Hough transform to perform straight line detection on the belt edge image to extract the belt edge line; and constructing the belt monitoring area based on the belt edge line.

[0008] Furthermore, the thermal image is preprocessed to extract the belt edge image, including: initializing the Gaussian kernel function as the spatial domain kernel function and the exponential kernel function as the range kernel function according to the size and pixel depth of the thermal image; using the spatial domain kernel function to perform weighted averaging on each pixel point of the thermal image and the pixel points in the corresponding neighborhood, updating the gray value of the current pixel point, and removing high-frequency noise; using the range kernel function to perform nonlinear combination of the weighted averaged gray value of the current pixel point and the original gray value to enhance the edge contour; iteratively executing the above weighted averaging and nonlinear combination steps until the preset convergence condition is reached, and outputting Denoised thermal imaging image; construct Sobel gradient operator convolution kernels in the X and Y directions according to the size of the thermal imaging image; perform horizontal convolution on the denoised thermal imaging image using the Sobel gradient operator convolution kernel in the X direction to obtain the horizontal gradient component Gx; perform vertical convolution on the denoised thermal imaging image using the Sobel gradient operator convolution kernel in the Y direction to obtain the vertical gradient component Gy; calculate the gradient amplitude G and gradient direction θ of each pixel based on the horizontal gradient component Gx and the vertical gradient component Gy to obtain the gradient amplitude map and the gradient direction map; extract the belt edge image based on the gradient amplitude map and the gradient direction map.

[0009] Furthermore, according to the gradient amplitude map and the gradient direction map, the belt edge image is extracted, including: initializing the gradient amplitude threshold T, marking the pixel points with gradient amplitude greater than T as candidate edge points, and marking the other pixel points as non-edge points, to obtain binary edge image data; according to the gradient direction data, quantizing 0°~360° into N discrete angles on average, dividing the candidate edge points into N discrete direction subsets, N is a positive integer greater than 4, to obtain gradient direction discretization data; traversing each discrete direction subset, and determining the sector to which the current candidate edge point belongs according to the gradient direction of the current candidate edge point; in the current discrete angle interval, searching for adjacent candidate edge points along the positive and negative gradient directions, and constructing a comparison unit for non-maximum suppression; comparing the gradient amplitudes of the current candidate edge point and its adjacent points, and deleting it from the binary edge image data if the gradient amplitude of the current point is not the maximum value in the comparison unit; iteratively traversing all discrete direction subsets, updating the binary edge image data, retaining the pixel points with the local maximum gradient amplitude in each direction, to obtain the belt edge image.

[0010] Preferably, according to the mean value of the gradient magnitude image data and standard deviation , adaptively calculate the high threshold and low threshold ; Where α and β are preset high and low threshold coefficients, and 0<β<α<1; traverse all pixel points of the binary edge image data, and mark them as strong edge points according to the relationship between their gradient amplitude G and the high and low thresholds , weak edge points and non-edge points , get the three-value edge image data; in the three-value edge image data, perform connectivity analysis on the 8-neighborhood of each weak edge point. If there is a strong edge point in the 8-neighborhood, merge the weak edge point into a strong edge point, otherwise delete it; iteratively analyze all weak edge points until only strong edge points and non-edge points are left in the three-value edge image, and output the final binary belt edge image data.

[0011] Furthermore, the belt edge image is detected by Hough transform to extract the belt edge line, including: establishing Hough parameter space, mapping the belt edge image data from the image space (xy) to the parameter space ; Traverse each foreground pixel point (x, y) in the belt edge image space, transform the pixel point into a parameter space curve, and perform Hough voting accumulation; extract the straight line parameters from the Hough voting accumulation data to obtain the belt edge line equation.

[0012] Furthermore, a Hough parameter space is established to map the belt edge image data from the image space (xy) to the parameter space , including: Based on the belt conveying direction, define an angle step of One-dimensional equal-division vector of , as the angular coordinate of the parameter space; calculate the scale factor between the image space and the parameter space , where H and W are the height and width of the belt edge image respectively; the parameter space is used to represent all possible straight lines in the image space, which is a two-dimensional space containing the distance of the straight line and angle Two parameters; based on the scale factor, the radius coordinate in parameter space is defined as , and follow the radius step Divide into Discrete sampling radius values; initialize the accumulator matrix in parameter space is a zero matrix with a matrix size of , each matrix element represents a parameter combination The corresponding straight line gets the number of votes, and the initial number of votes is 0.

[0013] Further, traverse each foreground pixel point (x, y) in the belt edge image space, transform the pixel point into the parameter space curve, and perform Hough voting accumulation, including: substituting the current pixel point (x, y) into the straight line parameter equation of the Hough transform , , , get a cosine curve in the parameter space; traverse each discretized sampling parameter point on the cosine curve , quantize it to the nearest accumulator matrix element , and add 1 to the value of the matrix element, thus completing the parameter point One vote; iteratively process all foreground pixels of the belt edge image, transform each edge pixel into a curve in the parameter space and vote, and finally obtain the complete Hough parameter space accumulator matrix .

[0014] Furthermore, the straight line parameters are extracted from the Hough voting cumulative data to obtain the belt edge line equation, including: using the imregionalmax function to calculate the accumulator matrix Perform regional maximum detection to find the maximum point with the most votes in the local neighborhood and form a maximum point set , where the number of maximum points K is determined by the threshold Control, only keep the number of votes The maximum point of The parameter coordinates are transformed back to the image space (xy) to obtain the equations of the K corresponding lines. ; Use the houghlines function to discretize the equation of each line and calculate the coordinates of the two intersection points between the line and the belt edge image boundary , and obtain K discretized edge segments represented by endpoints as the extracted belt edge lines.

[0015] Furthermore, a belt monitoring area is constructed based on the belt edge line, including: calculating the center line of the two belt edge lines as the symmetry axis of the belt; taking the center line as a reference, performing translation correction on the belt edge line to make it parallel to and equidistant from the center line; indenting the corrected belt edge line based on the standard width of the belt to obtain the inner edge line of the belt; connecting the endpoints of the two inner edge lines to form a closed belt monitoring area polygon.

[0016] Furthermore, an image analysis algorithm is used to delineate virtual restricted areas and virtual tripwires on the binocular image, and a violation database is established: a virtual restricted area is set, and two virtual tripwires are set on the boundary of the virtual restricted area as a dangerous area that users are prohibited from entering; a binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space, and generate a three-dimensional virtual tripwire containing height information, including: collecting a binocular image of the target area; using an image segmentation algorithm to perform feature analysis on the binocular image, identify and extract the target area in the image, and use the extracted target area as the virtual restricted area; according to the shape and size of the virtual restricted area, two parallel virtual tripwires are set at the boundary of the virtual restricted area; a binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space; the height information of the virtual tripwire is mapped to the set virtual tripwire, and a three-dimensional virtual tripwire containing height information corresponding to the position of the virtual tripwire is generated.

[0017] Furthermore, a binocular vision algorithm is used to calculate the height information of the virtual tripwire in the three-dimensional space, including: correcting the acquired binocular images, eliminating the distortion and position deviation between the left and right views, and aligning the left and right views to the same plane; detecting and extracting the pixel coordinates of the virtual tripwire in the corrected binocular images to obtain the two-dimensional coordinate information of the virtual tripwire in the left and right views; calculating the parallax of the pixel coordinates of the virtual tripwire in the left and right views, and combining the internal and external parameters of the binocular camera to calculate the height information of the virtual tripwire in the three-dimensional space through the triangulation principle.

[0018] Furthermore, during the production process, by analyzing the thermal imaging image, it is determined whether there is a target that meets the living body determination standard in the belt surface monitoring area, including: using an infrared thermal imager to collect the original infrared radiation signal of the belt surface monitoring area, the original infrared radiation signal includes the infrared radiation energy emitted by the target object in the monitoring area and the infrared radiation energy reflected from the background to the target object; according to the infrared radiation transmission distance and atmospheric conditions, the atmospheric attenuation coefficient is calculated, the original infrared radiation signal information is multiplied by the atmospheric attenuation coefficient, and the original infrared radiation signal is compensated for atmospheric attenuation to obtain an infrared radiation signal after atmospheric attenuation compensation; on the basis of the infrared radiation signal after atmospheric attenuation compensation, the infrared radiation value generated by the atmospheric temperature is superimposed to obtain a comprehensive radiation signal ; According to the lens material of the infrared thermal imager, the optical attenuation coefficient is calculated; the integrated radiation signal is multiplied by the optical attenuation coefficient, and the integrated radiation signal is optically attenuated to compensate for the optical attenuation, so as to obtain the infrared radiation signal after optical compensation; according to the infrared radiation signal after optical compensation, an infrared radiation data matrix is ​​generated through digital-to-analog conversion; according to the blackbody radiation equation, a mapping relationship between each element in the radiation data matrix and the corresponding temperature is established to obtain a two-dimensional temperature matrix reflecting the temperature distribution of the monitoring area; according to the two-dimensional temperature matrix, a thermal imaging map of the monitoring area is generated; the thermal imaging map is segmented to extract the connected areas with a temperature greater than a threshold; according to the preset living target judgment condition, it is judged whether the extracted connected areas with a temperature greater than a threshold meet the living target judgment condition.

[0019] Furthermore, in the production process, by analyzing the binocular images, it is determined whether the user has violated the rules and engaged in dangerous behaviors, including: using a binocular vision algorithm to calculate the three-dimensional spatial parameters of the moving object in the binocular image, and performing multi-frame statistical optimization and compensation: performing front and back frame differences on the collected continuous multi-frame binocular images to extract the binary contour image of the moving target; performing morphological filtering on the binary contour image to remove noise and holes to obtain a complete binary contour of the moving target; detecting the center of mass coordinates of the moving target in the binary contour images of the left and right views, calculating the parallax of the target center of mass, and combining the binocular camera parameters to calculate the three-dimensional spatial coordinates of the moving target through the triangulation principle; using Carl Zeiss's method to obtain a three-dimensional image of the moving target. The Mann filter algorithm filters the three-dimensional coordinate sequence of the moving target, estimates the target's state parameters such as speed and acceleration, and predicts the target position in the next frame; tracks the same moving target in multiple consecutive frames of images, and performs multi-frame cumulative averaging of its three-dimensional state parameters to reduce the single-frame measurement error; combines the three-dimensional position and trajectory of the moving target to determine whether it has entered the set warning area range, and issues a warning if it exceeds the range: compares the three-dimensional spatial position of the tracked moving target with the three-dimensional boundary model of the virtual restricted area; if the three-dimensional position of the moving target is within the boundary of the virtual restricted area, it is determined that the target has entered the warning area; according to the degree and duration of the target entering the warning area, The on-site users are warned and reminded by voice or image. When the moving target is detected to have crossed the height threshold set by the first three-dimensional virtual tripwire, it is judged as an illegal intrusion, and the sound and light alarm is triggered to remind the on-site personnel to pay attention to safety: the vertical distance of the moving target's center of mass relative to the first three-dimensional virtual tripwire is calculated in real time; if the vertical distance is less than the height threshold set by the three-dimensional virtual tripwire, and the target is within the projection boundary of the virtual tripwire, it is judged that the target has illegally invaded; the sound and light alarm device is linked to sound an alarm and flash a warning light to remind the on-site workers that someone has illegally entered the dangerous area and to pay attention to it; when the moving target is detected to continue to move toward the restricted area and cross the second three-dimensional virtual tripwire, Send out a locking control signal, cut off the power supply of the equipment, and force the operation to stop: track and analyze the three-dimensional displacement of the moving target for multiple consecutive frames to determine whether its movement direction is consistently towards the restricted area; calculate in real time the horizontal distance of the moving target's center of mass relative to the second three-dimensional virtual tripwire; if the moving target continues to move towards the restricted area and the horizontal distance between it and the second tripwire decreases to zero, it is determined that it has crossed the second tripwire; send a locking control command to the equipment control system to cut off the power supply of the operating equipment in the restricted area, forcing it to stop running to avoid harm to the intruders; at the same time, send out the highest level of sound and light alarm signal to remind all personnel to evacuate the dangerous area immediately and notify the on-site management personnel to deal with it.

[0020] Furthermore, the infrared thermal imager adopts an explosion-proof infrared thermal imager.

[0021] Another aspect of the present application also provides a system for detecting unsafe behaviors of personnel in an excavation working face, which is used to execute a method for detecting unsafe behaviors of personnel in an excavation working face of the present application.

[0022] Another aspect of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, a method for detecting unsafe behavior of personnel in an excavation working face is implemented.

[0023] Compared with the prior art, the advantages of this application are:

[0024] By simultaneously deploying infrared thermal imagers and visible light cameras in preset areas of the excavation working face, collecting thermal imaging images and binocular images, and fusing the image information of the two modalities to analyze and judge personnel behavior, the defect of insufficient judgment of a single information source is overcome, and all-round and multi-dimensional supervision of personnel behavior is achieved, effectively improving the pertinence and accuracy of hidden danger detection.

[0025] Image segmentation and edge detection algorithms are used to extract the belt area in the thermal imaging image and construct an accurate monitoring range. By setting temperature thresholds and area thresholds, live target detection and early warning are achieved in the monitoring area, and people appearing on the belt are found in time to prevent injuries caused by being caught in the belt or colliding with materials. Thermal imaging technology can work stably in harsh environments such as darkness and dust, ensuring the continuity and reliability of behavior monitoring.

[0026] Extract virtual restricted areas from binocular images and set up three-dimensional virtual tripwires. Use binocular vision's distance perception ability to determine whether a person has entered a dangerous area. Once a person is detected stepping on the line or entering a restricted area, an alarm is triggered, forming a "non-contact electronic fence" that can effectively prevent people from mistakenly entering a dangerous area. Converting the plane restricted area boundary into a three-dimensional tripwire containing height information improves the accuracy of spatial position discrimination and helps reduce the false alarm rate.

[0027] The original radiation signal collected by the thermal imager is subjected to a series of processing such as atmospheric attenuation compensation, optical attenuation compensation, and temperature mapping, and a living target discrimination model is constructed based on prior knowledge such as area and temperature. The precise positioning and identification of human heat sources is achieved in the complex underground environment, and the influence of high-temperature interference sources such as belts and machinery and equipment is eliminated, thus ensuring the reliability of living detection results.

[0028] The results of thermal imaging analysis and binocular image analysis are judged in time and space, and the position of the person detected by thermal imaging is used as a basis to judge whether the behavior of the person in the binocular image of the area is illegal. This breaks through the limitation of traditional behavior recognition algorithms that only focus on human movements and ignore spatial positions, and realizes a three-dimensional comprehensive judgment of "behavior + position", improving the comprehensiveness and accuracy of dangerous behavior identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0030] Figure 1 is a system architecture diagram of a method for detecting unsafe behaviors of personnel at a tunneling working face according to some embodiments of the present application;

[0031] Figure 2 is an image collected according to some embodiments of the present application;

[0032] Figure 3 is a schematic diagram of a system for detecting unsafe behaviors of personnel at a tunneling working face according to some embodiments of the present application;

[0033] Figure 4 is an exemplary flow chart of a method for detecting unsafe behavior of personnel at a tunneling working face according to some embodiments of the present application;

[0034] Figure 5 is a schematic diagram of a virtual tripwire according to some embodiments of the present application. DETAILED DESCRIPTION

[0035] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0036] A coal project applies the method of this application, such as Figure 1As shown. This system is designed and developed using the B / S (Browser / Server) architecture. The system is based on the Web browser / server mode and uses the browser to achieve comprehensive management and monitoring functions for coal conveying equipment. The system includes four layers: display layer, business layer, access layer, and database. Chip: RK3288 and RK3520 chips are used. Server: Intel Xeon processor (main frequency 2.6GHz) is configured and equipped with Redis cache to improve system performance. Software used Web server: Tomcat 8.0 is used, Java development kit: JDK 1.8 is selected, database: MySQL 5.6 database is used for data persistence, server operating system: CentOS 7.5. Display layer: Users can access the system through a Web browser on a computer or mobile phone. Access layer: HTTP protocol is used to receive requests from the display layer and pass the requests to the business layer for processing. Business layer: The core business logic of the system is deployed on the server, corresponding business processing is performed according to the request of the access layer, and the results are returned to the access layer. Database: MySQL database is used to store the business data and configuration information of the system. The business layer interacts with the database through JDBC and other methods to achieve data access. The system adopts a front-end and back-end separation deployment method. The back-end service is deployed on the CentOS server and runs through the Tomcat container. The front-end page accesses the back-end service interface through the HTTP protocol to obtain data and render the page. The system can flexibly expand the number of servers according to the number of visits and concurrency requirements. At the same time, Redis is used to cache hot data to improve the system response speed.

[0037] Image acquisition equipment is deployed on-site at a coal mine excavation face. The collected images are as follows: Figure 2 As shown. The excavation work area of ​​the excavation face is enclosed by a fence, and the fence encloses a dangerous area. In the case of double-tunnel excavation, the distance from the fence to the head does not exceed 150 meters. The width of the joint tunnel in the enclosed area is about 5.0 meters, the height is about 3.9 meters, and the length is about 19 meters. One EML360 continuous coal mining machine and one DSJ / 100 / 100 / 2×200 belt conveyor were put into use on site. The continuous coal mining machine is used to cut and load coal. Before each tunnel excavation, the continuous mining machine is first adjusted to the left side of the tunnel's forward direction, and the side laser and top laser are used for auxiliary positioning. The top laser line is 3.2 meters away from the left side of the tunnel, and the side laser line is 0.1 meters away from the tunnel side.

[0038] The continuous miner starts to cut the coal wall in front of it until the cutting depth reaches one cycle progress, that is, the grooving is completed. Then the continuous miner is withdrawn and adjusted to the right side of the tunnel to start cutting the remaining part, which is called stacking. The continuous miner gradually completes the tunnel excavation operation by alternating the grooving and stacking processes. In the coal loading link, the continuous miner adopts the self-loading method. While cutting coal, the continuous miner loads the fallen coal into the collection head. The collection head is equipped with disc rake claws, which continuously operate to load the coal into the middle conveyor of the continuous miner. The middle conveyor then transfers the coal to the shuttle car connected to the rear of the continuous miner to complete the collection and transfer of coal.

[0039] In the preset monitoring area of ​​the excavation working face, an infrared thermal imager and a visible light binocular camera are installed at the same time to cover the continuous miner operating area. The thermal imager collects thermal radiation information of the area and generates a thermal imaging image; the binocular camera collects synchronized color images and calculates depth of field information through binocular parallax. The system adopts the intelligent detection method of the present application, and uses temperature thresholds and area thresholds on the thermal imaging image to detect the heat source area of ​​the human body to determine whether there are people on the belt; a virtual restricted area and a three-dimensional tripwire are set on the color image to determine whether people have entered the dangerous area. Through the spatiotemporal correlation analysis of the detection results of thermal imaging and binocular images, violations are discovered in a timely manner and automatic alarms are issued, realizing real-time intelligent supervision of the behavior of personnel on the excavation working face.

[0040] like Figure 3 and Figure 4 As shown, a method for detecting unsafe behaviors of personnel in a tunneling working face comprises: installing an infrared thermal imager and a visible light camera in a preset area of ​​the tunneling working face: the infrared thermal imager is used to collect thermal imaging images on both sides of the tunneling machine to obtain infrared radiation signals of the target area; the visible light camera is used to collect binocular images on both sides of the tunneling machine to obtain visible light color and texture information of the target area; an image analysis algorithm is used to delineate a monitoring area of ​​a preset range of a belt on the thermal imaging image, and a temperature alarm threshold and a target area threshold are set; an image analysis algorithm is used to delineate a virtual restricted area and a virtual trip wire on the binocular image, and a violation database is established: a virtual restricted area is set, and two virtual trip wires are set on the boundary of the virtual restricted area; A virtual tripwire is used as a dangerous area where users are prohibited from entering. A binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space, and a three-dimensional virtual tripwire containing the height information is generated. During the production process, the thermal imaging image is analyzed to determine whether there is a target that meets the living judgment standard in the belt surface monitoring area. During the production process, the binocular image is analyzed to determine whether the user has violated regulations and engaged in dangerous behaviors. Based on the analysis results of the thermal imaging image and the binocular image, it is determined whether the user has engaged in unsafe behaviors: when the thermal imaging detects the presence of a living heat source in the belt monitoring area, and the binocular image analysis finds that the person is in a dangerous area or has made an illegal action, it is identified as an unsafe behavior.

[0041] Specifically, KAB24 (A) mining explosion-proof cameras are installed in preset areas of the tunneling working face, such as production areas or high-risk areas that need to be controlled. Infrared thermal imagers and visible light cameras are installed on both sides of the tunneling machine to ensure that image data of the target area can be collected. The infrared thermal imager collects thermal imaging images on both sides of the tunneling machine to obtain infrared radiation signals of the target area. Thermal imaging images are usually grayscale images, and each pixel value represents the temperature information of the point. The visible light camera collects binocular images on both sides of the tunneling machine to obtain visible light color and texture information of the target area. The binocular image includes two left and right perspectives, which can be used for subsequent three-dimensional reconstruction. The collected infrared thermal imaging images and visible light binocular images are transmitted to the data processing center via wired or wireless networks.

[0042] Using image analysis algorithm, the monitoring area of ​​the preset belt range is delineated on the thermal image, and the temperature alarm threshold and target area threshold are set. Specifically, for thermal image denoising, the Gaussian kernel function is initialized according to the size (width W and height H) and pixel bit depth of the thermal image. The standard deviation σ of the Gaussian kernel function can be set based on experience, for example, 1 / 10 of the image size. The size of the Gaussian kernel function can be set to , where k can be calculated based on the standard deviation σ, such as , Indicates rounding up. The element value of the Gaussian kernel function can be calculated by a two-dimensional Gaussian function, that is, , where (x, y) is the element coordinate. Normalize the Gaussian kernel function, , so that its elements sum to 1. According to the pixel bit depth, initialize the exponential kernel function. The form of the exponential kernel function can be set to , where x is the pixel gray value difference and h is the smoothing parameter. The smoothing parameter h can be set based on experience, for example, taking 1 / 10 of the pixel gray range. For each pixel point of the thermal imaging image, extract the The neighborhood window of the pixel. Multiply the Gaussian kernel function with the corresponding element of the neighborhood window to obtain the weighted coefficient. Multiply the pixel gray value in the neighborhood window with the corresponding weighted coefficient and sum them to obtain the weighted average gray value of the current pixel. . Update the gray value of the current pixel with the weighted average gray value Repeat the above steps until all pixels are traversed to obtain the image after spatial domain filtering.

[0043] For each pixel of the spatial domain filtered image, calculate the grayscale value difference x between it and the corresponding point in the original image. Substitute the grayscale value difference x into the exponential kernel function f(x) to obtain the weight coefficient w. Take the weighted average of the current pixel grayscale value of the spatial domain filtered image and the grayscale value of the corresponding point in the original image, that is, ,in is the gray value after nonlinear combination, is the gray value after spatial domain filtering, is the gray value of the original image, Represents the weight coefficient. Update the gray value of the current pixel with the gray value after nonlinear combination. Repeat the above steps until all pixels are traversed to obtain the image after range filtering. Use the image after range filtering as the new input and repeat. Set the number of iterations or convergence conditions, for example, the difference between the images of the previous and next iterations is less than a certain threshold. When the number of iterations or convergence conditions are reached, output the final denoised thermal imaging image.

[0044] Gradient calculation, construction of Sobel gradient operator convolution kernel: According to the size of the thermal imaging image, the Sobel gradient operator convolution kernel in the X and Y directions is constructed. The Sobel operator is a commonly used gradient operator, and its convolution kernel is a 3×3 matrix. The Sobel operator convolution kernel in the X direction is: , . Gradient in the X direction: ; Gradient in the Y direction: .

[0045] Calculate the horizontal gradient component :Use the X-direction Sobel gradient operator convolution kernel to perform horizontal convolution on the denoised thermal imaging image. The convolution operation can be performed by sliding the window, performing point multiplication between the convolution kernel and the local area of ​​the image and summing them. The convolution result is the horizontal gradient component , represents the gradient strength of the image in the X direction. Calculate the vertical gradient component :Use the Y-direction Sobel gradient operator convolution kernel to perform vertical convolution on the denoised thermal imaging image. Convolution operation and calculation Similar, but using the convolution kernel in the Y direction. The convolution result is the vertical gradient component , which represents the gradient strength of the image in the Y direction.

[0046] Calculate the gradient magnitude and direction: According to the horizontal gradient component and the vertical gradient component , calculate the gradient magnitude G and gradient direction θ of each pixel. The gradient magnitude G represents the gradient strength of the point and can be calculated by the following formula: The gradient direction θ represents the gradient direction of the point and can be calculated by the following formula: The calculated gradient magnitude G is assigned to the corresponding pixel point to obtain a gradient magnitude map. The calculated gradient direction θ is assigned to the corresponding pixel point to obtain a gradient direction map.

[0047] Belt edge extraction, initialize the gradient amplitude threshold T, which can be set based on experience or experiment. Traverse each pixel point of the gradient amplitude map, mark the pixel points with gradient amplitude greater than the threshold T as candidate edge points, and mark the remaining pixel points as non-edge points. Generate binary edge image data, the value of the candidate edge point is 1, and the value of the non-edge point is 0. According to the gradient direction data, the angle range of 0°~360° is quantized into N discrete angles on average, and N is a positive integer greater than 4. For example, 0°~360° can be divided into 8 sectors, each sector corresponding to an angle range of 45°. Divide the candidate edge points into corresponding discrete direction subsets according to their gradient direction θ, and obtain gradient direction discretization data. Traverse each discrete direction subset and process each candidate edge point in the subset. Determine the sector to which the current candidate edge point belongs according to its gradient direction θ. In the discrete angle interval corresponding to the current sector, search for adjacent candidate edge points along the positive and negative gradient directions to construct a comparison unit for non-maximum suppression. Compare the gradient amplitudes of the current candidate edge point with its adjacent points. If the gradient amplitude of the current point is not the maximum value within the comparison unit, delete it from the binary edge image data, i.e. mark it as a non-edge point. Iterate through all discrete direction subsets, update the binary edge image data, and retain the pixels with the local maximum gradient amplitude in each direction. Generate a belt edge image based on the updated binary edge image data. In the belt edge image, the value of the edge point is 1, and the value of the non-edge point is 0. Among them, the Sobel operator is used to calculate the horizontal and vertical gradients of the image, and the gradient amplitude and direction represent the edge strength and direction of the pixel points in the image. Binarization and non-maximum suppression are used to extract and refine the edges to obtain the final belt edge image.

[0048] Hough transform is used to detect the straight line of the belt edge image and extract the belt edge line; including: based on the belt conveying direction, an angle step is defined as One-dimensional equal-division vector of , as the angular coordinate of the parameter space. Calculate the scale factor between the image space and the parameter space , where H and W are the height and width of the belt edge image, respectively. The parameter space is a two-dimensional space containing two parameters, the distance ρ and the angle θ of the line, which are used to represent all possible lines in the image space. Based on the scale factor, the radius coordinate of the parameter space is defined as , and follow the radius step Divide into Discrete sampling radius values. Initialize the accumulator matrix of the parameter space is a zero matrix with a matrix size of , each matrix element represents a parameter combination The corresponding straight line gets the number of votes, and the initial number of votes is 0.

[0049] Traverse each foreground pixel point (x, y) in the belt edge image space, transform the pixel point to the parameter space curve, and perform Hough voting accumulation: Substitute the current pixel point (x, y) into the straight line parameter equation of the Hough transform ,in , θ∈[0, 180°], and obtain a cosine curve in parameter space. Traverse each discrete sampling parameter point on the cosine curve , quantize it to the nearest accumulator matrix element , and add 1 to the value of the matrix element, thus completing the parameter point Iterate all the foreground pixels of the belt edge image, transform each edge pixel into a curve in the parameter space and vote, and finally get the complete Hough parameter space accumulator matrix .

[0050] Extract the straight line parameters from the Hough voting cumulative data to obtain the belt edge line equation: Use the imregionalmax function to add the accumulator matrix Perform regional maximum detection to find the maximum point with the most votes in the local neighborhood and form a maximum point set , where the number of maximum points K is determined by the threshold Control, only keep the number of votes greater than According to the maximum point concentration, each maximum point The parameter coordinates are transformed back to the image space (xy) to obtain the equations of the K corresponding lines. The houghlines function is used to discretize the equation of each line and calculate the coordinates of the two intersection points between the line and the belt edge image boundary. , K discretized edge segments represented by endpoints are obtained as the extracted belt edge lines. Among them, Hough transform realizes straight line detection by mapping edge points in image space to curves in parameter space and accumulating votes in parameter space. The accumulator matrix records the number of votes for each parameter combination corresponding to the straight line. The higher the number of votes, the greater the possibility of the existence of the straight line. By finding the maximum point with the most votes in the accumulator matrix and inversely transforming it to the image space, the equation and discretized representation of the belt edge line are obtained.

[0051] According to the belt edge line, the belt monitoring area is constructed; including: calculating the belt center line: according to the two belt edge lines extracted, their center lines are calculated as the symmetry axis of the belt. Assume that the endpoint coordinates of the two belt edge lines are and The coordinates of the endpoints of the center line can be calculated by the average value of the corresponding endpoint coordinates, that is: The coordinates of the starting point of the center line: ; Centerline end point coordinates: ; Represent centerlines as line segments , as the symmetry axis of the belt.

[0052] Shift correction of belt edge line: Based on the center line, shift correction of belt edge line to make it parallel to the center line and at the same distance. Calculate the average distance between belt edge line and center line , which is half the belt width. For each belt edge line, calculate its vertical distance to the center line . Calculate the translation correction distance: According to the translation correction distance, the coordinates of the end point of the belt edge line are translated: The corrected starting point coordinates of the edge line: ;Corrected edge line end point coordinates: ; where θ is the angle of the center line, which can be obtained by Calculated. Get the belt edge line after translation correction , so that they are parallel to and equidistant from the center line.

[0053] Indented belt edge line: According to the standard width of the belt , shrink the corrected belt edge line inward to obtain the inner edge line of the belt. Calculate the shrinkage distance: . Perform inward correction on the coordinates of the end point of the belt edge line after translation correction: the coordinates of the starting point of the inner edge line: ; The coordinates of the end point of the inner edge line: ; Get the inner edge line of the belt after shrinking , whose width is equal to the standard belt width .

[0054] Construct the belt monitoring area polygon: connect the endpoints of the two inner edge lines to form a closed belt monitoring area polygon. The vertex coordinates of the belt monitoring area polygon are: , , , ; Connect these vertices in sequence to obtain a closed quadrilateral area as the belt monitoring area. Among them, by calculating the center line of the belt edge line, the symmetry axis of the belt is located; through translation correction and indentation operations, the inner edge line of the belt parallel to the center line and with the same width is obtained; finally, by connecting the endpoints of the inner edge line, a closed belt monitoring area polygon is formed.

[0055] like Figure 5As shown in the figure, a binocular camera is used to shoot the target area, and images from the left and right perspectives are obtained to form a binocular image pair. The left and right cameras of the binocular camera should have the same focal length, resolution, and optical axis parallelism, and the distance between the cameras should be fixed. The binocular image is processed using an image segmentation algorithm to divide the image into different regions: select a suitable image segmentation algorithm, such as threshold-based segmentation, edge detection segmentation, region growing segmentation, clustering segmentation, etc. Take the Graph Cut segmentation algorithm based on graph theory as an example: the image is represented as a weighted undirected graph , where the node V represents the pixel and the edge E represents the similarity between pixels. Define the energy function , where L is the pixel label. By minimizing the energy function, the optimal pixel label is solved and the segmentation result is obtained. The data term in the energy function The pixel p is marked as The cost of smoothing It measures the penalty for different labels of adjacent pixels p and q. It performs graph segmentation on the left and right views of the stereo camera to obtain different regions of the image.

[0056] Through feature extraction and pattern recognition technology, the target area in the image, such as pedestrians, vehicles, etc., is identified: in the segmented image area, the features of each area, such as color, texture, shape, etc. are extracted. Select appropriate feature descriptors, such as HOG (Histogram of Oriented Gradients) descriptors, SIFT (Scale-Invariant Feature Transform) descriptors, etc. Take the HOG descriptor as an example: divide the image into small cells, such as 8×8 pixels. Calculate the gradient direction histogram for each cell to obtain the HOG descriptor of the corresponding area. Combine adjacent cells into larger blocks and normalize the HOG descriptors within the block. Concatenate the HOG descriptors of all blocks to obtain the HOG feature vector of the entire image. Use machine learning algorithms to classify and identify the extracted features, such as SVM (Support Vector Machine), decision trees, neural networks, etc. When training the classifier, use the manually annotated target areas (such as pedestrians and vehicles) as positive samples and other areas as negative samples. Apply the trained classifier to the test image to identify the target area in the image.

[0057] Extract the identified target area as a virtual restricted area: For the identified target area, extract its contour boundary to obtain a closed polygonal area. Use the extracted polygonal area as a virtual restricted area to indicate an area that is not allowed to enter or cross. The attributes of the virtual restricted area include position, size, shape, etc., which can be represented by a sequence of polygon vertex coordinates. According to the shape and size of the virtual restricted area, set two parallel virtual tripwires on its boundary: Analyze the shape characteristics of the virtual restricted area polygon, such as length, width, direction, etc. According to the shape of the virtual restricted area, select appropriate positions on its boundary to set two parallel virtual tripwires. The direction of the virtual tripwire should be consistent with the main direction of the virtual restricted area, and the length should cover the main part of the restricted area. The distance between the two virtual tripwires should be set according to actual needs and scenarios, and should fully cover the restricted area without being too dense. Represent the virtual tripwire as the starting and end coordinates , or the parameters (a, b, c) of the straight line equation ax+by+c=0, calculate the area A of the virtual restricted area: ,in, , that is, the area of ​​the virtual restricted area is calculated by the polygon vertices. The perimeter P of the virtual restricted area is: .

[0058] Correct the acquired binocular images: Use Zhang's calibration method to calibrate the binocular camera to obtain the camera's intrinsic and extrinsic parameters. The intrinsic parameters include focal length , principal point coordinates , distortion coefficient The external parameters include the rotation matrix R and the translation vector T, which represent the relative position relationship between the left and right cameras. The Bouguet algorithm is used to correct the binocular image: According to the internal and external parameters obtained by calibration, the correction transformation matrix of the left and right cameras is calculated. and the projection matrix For each pixel point (x, y) in the left and right views, the correction transformation matrix is ​​used to map it to the corrected pixel point (Left view); (Right view); According to the corrected pixel coordinates, the left and right views are resampled to obtain the corrected binocular image. In the corrected binocular image, the corresponding points are on the same line, eliminating distortion and position deviation.

[0059] Detect and extract the pixel coordinates of the virtual tripwire in the rectified binocular image: Perform edge detection on the rectified left and right views, such as using the Canny algorithm. In the edge image, use the Hough transform to detect straight lines and obtain the pixel coordinates of the virtual tripwire. The Hough transform maps the image space (x, y) to the parameter space , where ρ is the distance from the line to the origin, and θ is the angle of the line. The equation of a line can be expressed as: For each edge point (x, y) in the image, there is a corresponding sine curve in the parameter space. The intersection of multiple sine curves corresponds to a straight line in the image space. The value is the parameter of the line. Get the two-dimensional coordinates of the corresponding points of the virtual trip wire in the left view and the right view and .

[0060] Calculate the disparity of the virtual tripwire pixel coordinates in the left and right views: Since the corresponding points in the corrected binocular image are on the same line, the disparity The parallax is inversely proportional to the distance of the object from the camera: , where f is the focal length, B is the baseline distance, and Z is the depth of the object. Combining the internal and external parameters of the binocular camera, the three-dimensional coordinates of the virtual tripwire are calculated by the triangulation principle: According to the corrected pixel coordinates and , calculate the depth of the object point . Using the projection matrix and , convert pixel coordinates and depth into three-dimensional coordinates (X, Y, Z) of the object point: (Left view); Right view); obtain the height information Z of the virtual trip wire in three-dimensional space.

[0061] Generate a three-dimensional virtual tripwire containing height information: map the height information Z of each point on the virtual tripwire to the set virtual tripwire. Generate a three-dimensional virtual tripwire according to the two-dimensional position (x, y) of the virtual tripwire and the corresponding height information Z. The three-dimensional virtual tripwire can be represented as a collection of a series of three-dimensional coordinate points (X, Y, Z) to form a three-dimensional straight line. Binocular correction eliminates image distortion and position deviation, edge detection and Hough transform extract the pixel coordinates of the virtual tripwire, and parallax calculation and triangulation principles obtain the three-dimensional coordinates and height information of the virtual tripwire. In the production process, by analyzing the thermal imaging image, it is determined whether there is a target that meets the living judgment standard in the belt surface monitoring area; including: selecting a suitable infrared thermal imager, such as an uncooled focal plane array infrared thermal imager. Install the infrared thermal imager above the belt surface monitoring area, adjust the focal length and field of view to ensure that the entire monitoring area can be covered. The infrared thermal imager receives the infrared radiation energy emitted by the target object in the monitoring area and the infrared radiation energy reflected by the background to the target object, and obtains the original infrared radiation signal. The atmospheric attenuation coefficient τ is calculated based on the infrared radiation transmission distance and atmospheric conditions (such as temperature, humidity, air pressure, etc.). The atmospheric attenuation coefficient τ can be calculated using the MODTRAN atmospheric transmission model or empirical formula.

[0062] The original infrared radiation signal Multiplying it with the atmospheric attenuation coefficient τ, we get the infrared radiation signal after atmospheric attenuation compensation. : According to the atmospheric temperature , calculate the infrared radiation value generated by the atmosphere's own radiation The atmosphere's own radiation can be calculated using Planck's blackbody radiation formula: , where σ is the Stefan-Boltzmann constant The infrared radiation signal after atmospheric attenuation compensation Radiation from the atmosphere itself Add together to get the comprehensive radiation signal : . Calculate the optical attenuation coefficient κ based on the lens material of the infrared thermal imager, such as germanium, silicon, calcium fluoride, etc. The optical attenuation coefficient κ is related to the transmittance of the lens material and can be obtained by consulting the material manual or experimental measurement.

[0063] The integrated radiation signal Multiplying by the optical attenuation coefficient κ, we get the infrared radiation signal after optical compensation. : The focal plane array detector of the infrared thermal imager receives the infrared radiation signal after optical compensation. , convert it into an electrical signal. The electrical signal is converted into a digital signal through analog-to-digital conversion (ADC) to obtain the infrared radiation data matrix . Infrared Radiation Data Matrix Each element in represents the infrared radiation intensity received at the corresponding pixel position. According to Planck's blackbody radiation formula, the relationship between radiation intensity I and temperature T is established: , where ε is the emissivity of the target object. For the radiation data matrix For each element in, calculate the corresponding temperature value according to the above formula to obtain a two-dimensional temperature matrix . Two-dimensional temperature matrix It reflects the temperature distribution in the monitoring area. Map to grayscale or pseudo color to generate thermal images Thermal imaging images The temperature distribution of the monitored area is intuitively displayed. The higher the temperature, the brighter the color or the specific color. Set the temperature threshold , the thermal image Binarization, get a binary image : ; For binary images Perform connected area analysis and extract the areas with temperature greater than the threshold The connected area . Preset living target judgment conditions, such as area, aspect ratio, temperature average, etc. For each extracted connected area , calculate its area , aspect ratio , mean temperature And other characteristic parameters.

[0064] Determine whether the characteristic parameters meet the preset living target judgment conditions: If and and but Living target; judgement in other cases for non-living targets; according to the judgment results, determine whether there are living targets in the monitoring area. The infrared thermal imager collects the original infrared radiation signal, obtains the optically compensated infrared radiation signal after atmospheric attenuation compensation, atmospheric radiation superposition and optical attenuation compensation, and then generates the infrared radiation data matrix and two-dimensional temperature matrix through digital-to-analog conversion to generate thermal imaging images. Perform image segmentation and connected area analysis on the thermal imaging image, and judge whether the extracted high-temperature area is a living target according to the preset living target judgment conditions. It can effectively detect abnormally high-temperature objects on the belt surface and provide important information for accident prevention.

[0065] During the production process, binocular images are analyzed to determine whether the user has violated regulations or engaged in dangerous behaviors; explosion-proof infrared thermal imaging cameras for mining are installed on both sides of the tunnel boring machine (safety stop distance). During the production process, if a hot object is found passing through the monitoring area with a temperature higher than 30 degrees and an area reaching the set pixel point, the system regards it as a living object, and the platform sends a voice alarm signal to remind the on-duty personnel. At the same time, the intelligent analysis device sends a current signal, which is transmitted to the on-site PLC system and the intelligent analysis device in the control room through a multi-functional optical terminal, generating an alarm stop signal. Including:

[0066] The binocular vision algorithm is used to calculate the three-dimensional spatial parameters of the moving object in the binocular image, and multi-frame statistical optimization and compensation are performed: the front and back frames of the collected continuous multi-frame binocular images are differentiated to extract the binary contour image of the moving target: the continuous frame images of the left and right views are and Perform grayscale and Gaussian filtering to obtain a smoothed grayscale image and .right and Perform frame difference calculation to obtain the differential image : . For the difference image Perform binarization processing to obtain the binary contour image of the moving target : ; Perform morphological filtering on the binary contour image to remove noise and holes and obtain a complete binary contour of the moving target: Perform morphological opening operations to remove small noise points: , S is the structural element. Express Perform morphological closing operation, This means filling holes inside a target: .

[0067] The centroid coordinates of the moving target are detected in the binary contour images of the left and right views, the parallax of the target centroid is calculated, and the three-dimensional spatial coordinates of the moving target are calculated by triangulation principle combined with the binocular camera parameters: and Perform connected area analysis to extract the centroid coordinates of the moving target and . Calculate the disparity of the target center of mass in the left and right views . Combined with the baseline distance B, focal length f, and pixel size of the binocular camera , calculate the depth of the target centroid through the triangulation formula . According to the center of mass coordinates and depth, the three-dimensional space coordinates (X, Y, Z) of the target center of mass are calculated.

[0068] The Kalman filter algorithm is used to filter the three-dimensional coordinate sequence of the moving target, estimate the target's speed and acceleration and other state parameters, and predict the target position in the next frame: establish the state vector of the moving target , including position and velocity. According to the state estimation value of the previous frame and the observation value of the current frame , update the state estimate of the current frame through the Kalman filter formula . Use the state transfer matrix F to predict the target state vector of the next frame Track the same moving target in multiple consecutive frames, and perform multi-frame cumulative averaging of its 3D state parameters to reduce the single-frame measurement error: Use the 3D coordinates and motion state of the target's center of mass to establish a matching relationship between the previous and next frames to achieve target tracking. Cumulative averaging of multi-frame state parameters of the same target being tracked: , reducing the error.

[0069] Determine whether the mobile target has entered the set warning area: compare the three-dimensional spatial position (X, Y, Z) of the tracked mobile target with the pre-set three-dimensional geometric boundary model of the virtual restricted area. The three-dimensional geometric boundary model of the virtual restricted area can be created based on the on-site equipment layout and safety requirements using three-dimensional modeling technology in computer graphics, such as three-dimensional grid models, CSG models, etc. To determine whether the three-dimensional coordinate point (X, Y, Z) of the mobile target is located inside the boundary of the virtual restricted area, a point-polygon internal test algorithm can be used, such as the ray method, angle sum method, etc. If the three-dimensional coordinate point of the mobile target is inside the boundary of the virtual restricted area, it is determined that the target has entered the warning area. Take corresponding warning measures based on the degree and duration of the target's entry into the warning area: calculate the shortest distance between the current position of the target and the boundary of the virtual restricted area. , the shortest distance algorithm from point to polygon can be used. Calculate the duration of the target entering the restricted area , which is the time difference from the target entering the restricted area boundary to the current moment. Less than the set distance threshold (such as 0.5 meters) and If the time is greater than the set threshold (such as 2 seconds), a warning signal is triggered. A voice prompt is generated through speech synthesis technology, such as "Warning, someone has entered the danger zone, please stay away!". A warning box pops up on the monitoring interface through image display technology, and the target location is marked.

[0070] For illegal intrusion judgment and alarm when the moving target crosses the first three-dimensional virtual tripwire, the vertical distance of the moving target's center of mass relative to the first three-dimensional virtual tripwire is calculated in real time: according to the endpoint coordinates of the first three-dimensional virtual tripwire and , calculate the height range of the tripwire . , indicating the minimum height of the tripwire. , represents the highest height of the tripwire. Calculate the vertical distance from the moving target's center of mass coordinates (X, Y, Z) to the plane where the first tripwire is located : The plane where the tripwire is located can be represented by a normal vector and a point To express.

[0071] Normal vector It can be calculated by the coordinate difference of the two tripwire endpoints: ; ; ; Normalize the normal vector by dividing it by its modulus , , , , the vertical distance from the target center of mass coordinate to the plane where the trip wire is located The distance from a point to a plane can be calculated using the distance formula: ; Determine whether the target centroid coordinates (X, Y) are within the projection boundary of the first virtual tripwire: and Project it onto the XY plane to form a two-dimensional line segment. Use the point-to-line segment distance formula to calculate the distance from the target centroid (X, Y) to the two-dimensional line segment. If the distance If it is smaller than a set threshold (such as 0.1 meters), the target center of mass is considered to be within the projected boundary of the tripwire.

[0072] Determine if the target has violated the rules and trigger the sound and light alarm: If the vertical distance from the target's center of mass to the trip wire is Less than the set height threshold , and the target's center of mass is within the projection boundary of the tripwire, then the target is judged to have violated the rules. Link the sound and light alarm device, and control the alarm and warning light through the relay or PLC: sound an alarm to alert the on-site workers. The alarm can be a pre-recorded audio file, or it can be generated in real time using audio synthesis technology. Flash the warning light to attract the visual attention of on-site personnel. The warning light can use an LED light or a rotating warning light, and the flashing effect can be achieved through the control circuit. A striking alarm prompt box pops up on the monitoring interface, displaying information such as "Someone has illegally entered the dangerous area, please deal with it immediately". At the same time, the illegal intrusion event is recorded in the log file or database, including the time, location, target ID and other information of the event, to facilitate subsequent analysis and auditing.

[0073] If no personnel manually eliminate the alarm within the set time (such as 30 seconds), the alarm information will be automatically reported to the superior management system or the on-duty personnel, and notifications will be sent through SMS, email, etc. By calculating the spatial relationship between the target center of mass and the trip wire in real time, and setting a reasonable vertical distance and projection boundary threshold, the target's violation behavior can be accurately judged. At the same time, the sound and light alarm device is linked to prompt the on-site personnel to pay attention in the form of sound and light, and the management personnel are notified through the reporting mechanism, so that illegal behaviors can be stopped and handled in time, effectively reducing safety hazards in coal mines.

[0074] Lock control and alarm when the moving target continues to move towards the restricted area and crosses the second 3D virtual trip wire. Track and analyze the 3D displacement of the moving target in multiple frames to determine whether its movement direction is consistent in the restricted area: Extract the 3D center of mass coordinates of the moving target from the continuous multi-frame tracking results. , where i represents the frame number. Calculate the three-dimensional displacement vector of the moving target in multiple consecutive frames : , , ; Calculate the restricted area direction vector :According to the 3D geometric model of the restricted area, select two points on the boundary of the restricted area and The restricted area direction vector can be expressed as: , , ; Normalize the restricted area direction vector by dividing it by its modulus . Calculate the displacement vector and the restricted area direction vector The angle θ is: ; If the angle θ is less than a set angle threshold (such as 30°), it is determined that the moving target is moving toward the restricted area.

[0075] Calculate the horizontal distance of the moving target's center of mass relative to the second three-dimensional virtual tripwire in real time: according to the endpoint coordinates of the second three-dimensional virtual tripwire and , calculate the horizontal direction vector of the tripwire : , ; Calculate the vertical distance from the moving target's center of mass (X, Y, Z) on the horizontal plane to the second tripwire : Project the target center of mass coordinates and the tripwire endpoint coordinates onto the XY plane to obtain (X, Y), and . Using the distance formula from a point to a line, calculate the vertical distance from (X, Y) to the tripwire If the moving target continues to move towards the restricted area, and When it decreases to zero, the target is judged to have crossed the second tripwire.

[0076] Send a locking control command, cut off the power supply of the equipment, and force the operation to stop: Send a locking control command to the equipment control system and transmit it through the digital IO interface or industrial bus protocol (such as Modbus, Profibus). The control command includes information such as device ID, operation code (such as turning off the power), timestamp, etc. to ensure the accuracy and timeliness of the command. After receiving the locking command, the equipment control system immediately cuts off the power supply of the operating equipment in the restricted area and forces it to stop running. At the same time, the equipment status information is fed back to the upper computer to confirm that the equipment has stopped to avoid misoperation.

[0077] Send out the highest level of sound and light alarm signals to remind people to evacuate: Trigger the highest level of sound and light alarm devices, such as sirens, flashing lights, etc., to emit harsh alarm sounds and eye-catching flashing signals. Play pre-recorded warning messages through the voice broadcast system, such as "Warning! Someone has entered the dangerous restricted area, all personnel evacuate immediately! The equipment is about to shut down!". The highest level of alarm prompt box pops up on the monitoring interface, displaying the alarm information and the scene picture to attract the attention of the on-duty personnel. Automatically send SMS, email and other notifications to the on-site management personnel to inform the severity and urgency of the incident and require immediate disposal. By tracking the movement trend of the target in real time, judging whether it continues to move towards the restricted area, and calculating its distance from the key trip wire, high-risk intrusion behaviors can be accurately identified. When the highest level of sound and light alarm is triggered, the power supply of the equipment is immediately cut off and the operation is forced to stop, which can effectively avoid collisions between intruders and equipment and prevent accidental injuries. At the same time, through a variety of notification methods, quickly inform all personnel on site to evacuate, and require management personnel to deal with it in time to maximize the safety of the life of the operating personnel.

[0078] According to the analysis results of thermal imaging images and binocular images, it is judged whether the user has unsafe behavior: if the thermal imaging detects the presence of a living heat source in the belt monitoring area, and the binocular image analysis finds that the person is in a dangerous area or makes an illegal action, it is considered an unsafe behavior. Generate alarm information for unsafe behavior, including the time, location, person ID, specific behavior description, etc. of the incident, and submit it to the upper-level safety management system. Establish a visualization sample library of key unsafe behaviors in dangerous areas of the excavation working face: collect actual operation video and image data in dangerous areas of the excavation working face as the basis for analysis and modeling. Manually annotate the collected data to identify the human body area, dangerous area, unsafe behavior, etc. in the image and video. Perform statistical analysis on the annotated data to summarize the unsafe behavior characteristics of the operators, such as high-frequency illegal actions and typical illegal postures. Prepare a standard template for high-risk unsafe behavior characteristics, and describe in detail the action elements, danger level, consequence analysis, etc. of each unsafe behavior. Use the annotated data and feature standards to establish a visualization sample library of key unsafe behaviors in dangerous areas of the excavation working face. The sample library contains a large number of typical unsafe behavior images and video clips, and each sample comes with a detailed behavior label and description. Use data enhancement techniques, such as image flipping, rotation, and scaling, to expand the size and diversity of the sample library. Classify and organize the sample library, and organize and index it according to the type and danger level of unsafe behaviors. Use the visual sample library to compare and analyze newly collected operation videos and images. Use content-based image retrieval techniques, such as feature matching and similarity measurement, to search the sample library for unsafe behavior samples similar to new images. Visually display the matching results to help safety managers quickly identify and locate unsafe behaviors, and conduct risk assessments and early warnings.

Claims

1. A method for detecting unsafe behavior of personnel at a tunneling face, characterized in that: include: Install infrared thermal imagers and visible light cameras in the preset area of ​​the tunneling working face. The infrared thermal imagers are used to collect thermal imaging images on both sides of the tunneling machine and obtain infrared radiation signals of the target area. The visible light camera is used to collect binocular images on both sides of the tunnel boring machine to obtain visible light color and texture information of the target area; Using image analysis algorithms, the monitoring area of ​​the preset belt range is delineated on the thermal imaging image, and the temperature alarm threshold and target area threshold are set; The image analysis algorithm is used to delineate virtual restricted areas and virtual tripwires on the binocular image, and a violation database is established. A virtual restricted area is set up, and two virtual tripwires are set up on the boundary of the virtual restricted area as a dangerous area that users are prohibited from entering. The binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space, and a three-dimensional virtual tripwire containing height information is generated. During the production process, thermal imaging images are analyzed to determine whether there are targets that meet the living body judgment criteria in the belt surface monitoring area; During the production process, the binocular images are analyzed to determine whether the user has violated regulations or engaged in dangerous behavior; Based on the analysis results of the thermal imaging image and the binocular image, it is determined whether the user has engaged in unsafe behavior. When the thermal imaging detects the presence of a living heat source in the belt monitoring area, and the binocular image analysis finds that the person is in a dangerous area or has made an illegal action, it is considered an unsafe behavior. Using image analysis algorithms, the monitoring area of ​​the belt preset range is delineated on the thermal imaging image, including: Preprocess the thermal image to extract the belt edge image; Hough transform is used to detect straight lines on the belt edge image and extract the belt edge line; Construct the belt monitoring area according to the belt edge line; Preprocess the thermal image to extract the belt edge image, including: According to the size and pixel depth of the thermal imaging image, the Gaussian kernel function is initialized as the spatial domain kernel function, and the exponential kernel function is initialized as the value range kernel function; The spatial domain kernel function is used to perform weighted averaging on each pixel of the thermal imaging image and the pixels in the corresponding neighborhood, update the gray value of the current pixel, and remove high-frequency noise; The range kernel function is used to nonlinearly combine the weighted average grayscale value of the current pixel with the original grayscale value to enhance the edge contour; Iteratively execute the above weighted average and nonlinear combination steps until the preset convergence condition is reached, and output the denoised thermal imaging image; According to the size of the thermal imaging image, the Sobel gradient operator convolution kernel in the X and Y directions is constructed; The denoised thermal imaging image is horizontally convolved using the X-direction Sobel gradient operator convolution kernel to obtain the horizontal gradient component Gx; The denoised thermal imaging image is vertically convolved using the Sobel gradient operator convolution kernel in the Y direction to obtain the vertical gradient component Gy; According to the horizontal gradient component Gx and the vertical gradient component Gy, the gradient amplitude G and the gradient direction θ of each pixel are calculated to obtain the gradient amplitude map and the gradient direction map; The belt edge image is extracted based on the gradient amplitude map and the gradient direction map.

2. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 1, characterized in that: Hough transform is used to detect straight lines on the belt edge image and extract the belt edge line, including: Establish Hough parameter space and map the belt edge image data from image space (xy) to parameter space ; Traverse each foreground pixel (x, y) in the belt edge image space, transform the pixel into the parameter space curve, and perform Hough voting accumulation; The straight line parameters are extracted from the Hough voting accumulation data to obtain the belt edge line equation.

3. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 2, characterized in that: Establish Hough parameter space and map the belt edge image data from image space (xy) to parameter space ,include: Based on the belt conveying direction, define an angle step size of One-dimensional equal-division vector of , as the angular coordinate in the parameter space; Calculate the scale factor between image space and parameter space , where H and W are the height and width of the belt edge image respectively; the parameter space is used to represent all possible straight lines in the image space, which is a two-dimensional space containing the distance of the straight line and angle Two parameters; Based on the scale factor, the radius coordinate in the parameter space is defined as , and follow the radius step Divide into discrete sampling radius values; Initialize the accumulator matrix in parameter space is a zero matrix with a matrix size of , each matrix element represents a parameter combination The corresponding straight line gets the number of votes, and the initial number of votes is 0.

4. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 3, characterized in that: Traverse each foreground pixel (x, y) in the belt edge image space, transform the pixel into the parameter space curve, and perform Hough voting accumulation, including: Substitute the current pixel point (x, y) into the line parameter equation of the Hough transform ,in, , , and obtain a cosine curve in parameter space; Traverse each discretized sampling parameter point on the cosine curve , quantize it to the nearest accumulator matrix element , and add 1 to the value of the matrix element, thus completing the parameter point A vote of Iteratively process all foreground pixels of the belt edge image, transform each edge pixel into a curve in the parameter space and vote, and finally obtain the complete Hough parameter space accumulator matrix .

5. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 4, characterized in that: Extract the straight line parameters from the Hough voting cumulative data to obtain the belt edge line equation, including: For the accumulator matrix Perform regional maximum detection to find the maximum point with the most votes in the local neighborhood and form a maximum point set , where the number of maximum points K is determined by the threshold Control, only keep votes greater than the threshold The maximum point of ; According to the maximum point concentration each maximum point The parameter coordinates are transformed back to the image space (xy) to obtain the equations of the K corresponding lines: ; The houghlines function is used to discretize the equation of each line and calculate the coordinates of the two intersection points between the line and the belt edge image boundary. , and obtain K discretized edge segments represented by endpoints as the extracted belt edge lines.

6. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 5, characterized in that: According to the belt edge line, the belt monitoring area is constructed, including: Calculate the center line of the two belt edge lines as the belt's axis of symmetry; Taking the center line as the reference, perform translation correction on the belt edge line to make it parallel to the center line and at the same distance; According to the standard width of the belt, the corrected belt edge line is retracted to obtain the inner edge line of the belt; Connect the endpoints of the two inner edge lines to form a closed belt monitoring area polygon.

7. The method for detecting unsafe behaviors of personnel in a tunneling working face according to any one of claims 1 to 6, characterized in that: Image analysis algorithms are used to define virtual restricted areas and virtual tripwires on binocular images, including: Set up a virtual restricted area, and set up two virtual tripwires on the boundary of the virtual restricted area as a dangerous area that users are prohibited from entering; Using binocular vision algorithm, the height information of the virtual tripwire in three-dimensional space is calculated to generate a three-dimensional virtual tripwire containing height information, including: Collect binocular images of the target area; The image segmentation algorithm is used to analyze the features of the binocular image, identify and extract the target area in the image, and use the extracted target area as a virtual restricted area; According to the shape and size of the virtual restricted area, two parallel virtual tripwires are set at the boundary of the virtual restricted area; The binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space; The height information of the virtual tripwire is mapped to the set virtual tripwire to generate a three-dimensional virtual tripwire containing height information corresponding to the position of the virtual tripwire.

8. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 7, characterized in that: The binocular vision algorithm is used to calculate the height information of the virtual tripwire in three-dimensional space, including: Correct the acquired binocular images to eliminate the distortion and position deviation between the left and right views, and align the left and right views to the same plane; Detect and extract the pixel coordinates of the virtual tripwire in the calibrated binocular image to obtain the two-dimensional coordinate information of the virtual tripwire in the left and right views; The parallax of the pixel coordinates of the virtual tripwire in the left and right views is calculated, and the height information of the virtual tripwire in three-dimensional space is calculated by the triangulation principle in combination with the internal and external parameters of the binocular camera.

9. The method for detecting unsafe behavior of personnel at a tunneling face according to claim 8, characterized in that: During the production process, thermal imaging images are analyzed to determine whether there are targets that meet the living body judgment criteria in the belt surface monitoring area, including: The original infrared radiation signal of the belt surface monitoring area is collected by using an infrared thermal imager. The original infrared radiation signal includes the infrared radiation energy emitted by the target object in the monitoring area and the infrared radiation energy reflected from the background to the target object; According to the infrared radiation transmission distance and atmospheric conditions, the atmospheric attenuation coefficient is calculated, the original infrared radiation signal information is multiplied by the atmospheric attenuation coefficient, the original infrared radiation signal is compensated for atmospheric attenuation, and the infrared radiation signal after atmospheric attenuation compensation is obtained; On the basis of the infrared radiation signal after atmospheric attenuation compensation, the infrared radiation value generated by the atmospheric temperature is superimposed to obtain a comprehensive radiation signal; According to the lens material of the infrared thermal imager, the optical attenuation coefficient is calculated; the integrated radiation signal is multiplied by the optical attenuation coefficient, and optical attenuation compensation is performed on the integrated radiation signal to obtain an infrared radiation signal after optical compensation; According to the infrared radiation signal after optical compensation, an infrared radiation data matrix is ​​generated through digital-to-analog conversion; According to the blackbody radiation equation, a mapping relationship between each element in the radiation data matrix and the corresponding temperature is established to obtain a two-dimensional temperature matrix reflecting the temperature distribution of the monitoring area; Generate a thermal image of the monitoring area based on the two-dimensional temperature matrix; Perform image segmentation on the thermal image and extract the connected areas with temperature greater than the threshold; According to the preset living object judgment condition, it is determined whether the connected area whose extracted temperature is greater than the threshold satisfies the living object judgment condition.

10. The method for detecting unsafe behaviors of personnel at a tunneling working face according to claim 9, characterized in that: The infrared thermal imager uses an explosion-proof infrared thermal imager.

Images