Intelligent identification method for dangerous area invaded by personnel on fully mechanized coal mining face of coal mine

Abstract

The invention discloses an intelligent identification method for personnel invading a dangerous area in a fully mechanized coal mining face of a coal mine. The method comprises the following steps: S1, collecting video image data of personnel in the fully mechanized coal mining face, a face guard and a coal baffle; s2, video image defogging and enhancement preprocessing; s3, establishing a personnel intrusion identification model based on an improved YOLOv8s target detection method; s4, automatically dividing a personnel invasion dangerous area; and S5, identifying the personnel invading the dangerous area. According to the method, aiming at a to-be-detected coal mine specific scene, through feature detection and adaptive drawing of the dangerous area, automatic and flexible division of the dangerous area is realized, the adaptability is high, accurate identification and advanced pre-control of personnel invasion in the dangerous area of a coal mine fully mechanized coal mining face can be realized, the mine production efficiency and safety protection are improved, and the economic benefit is high. The intelligent construction of the fully mechanized coal mining face is promoted, and the life safety of miners is effectively guaranteed.

Description

Technical Field

[0001] The invention belongs to the technical field of coal mine safety, and in particular relates to an intelligent identification method for personnel intrusion into a dangerous area on a fully-mechanized mining face in a coal mine. Background Art

[0002] With the introduction of technologies such as artificial intelligence, 5G communications, and big data, the production work of fully mechanized coal mining faces is developing in the direction of automation and intelligence, striving to achieve remote monitoring and intelligent decision-making and automated collaborative operation of various systems to ensure the safe and efficient operation of fully mechanized coal mining faces. In fully mechanized coal mining faces, when the scraper conveyor is running, the area between its coal baffle and the coal wall is the working area of ​​the coal mining machine and the scraper, and personnel entering by mistake will cause inevitable accidents. Traditional identification methods have certain limitations. Therefore, it is urgent to study the intelligent identification of personnel in fully mechanized coal mining faces and solve the problem of personnel safety.

[0003] For dangerous areas in mines, the safety factor of traditional fences is not high, and the area is fixed, with poor flexibility and timeliness. For traditional video surveillance, although it can intelligently complete the acquisition of information and storage of data, it is unable to perform corresponding analysis and processing on the acquired information, and the missed detection rate is high if it relies solely on manual observation and identification. At present, there are also related studies on personnel intrusion detection in coal mines based on deep learning. One is to directly arrange cameras in the target detection area, and directly detect whether there are human targets through the camera to judge human intrusion; the other is to draw a warning line or warning area to trip wire or cross-border alarm to detect the status of human intrusion. However, this method has a fixed area and low flexibility. When the background area picture shot by the camera jumps or rotates changes, it is easy to make misjudgments and false alarms, and the adaptability to scene changes is poor. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for intelligently identifying dangerous areas of personnel intrusion in a coal mine fully-mechanized working face in view of the deficiencies in the above-mentioned prior art, which makes the target contour information clearer and improves the detection accuracy through image defogging and enhancement processing; stores the coordinate information of the detection target anchor frame and sorts it in a specific method to extract the coordinate information of the key points of the detection target; connects the key points to take the intersection with the image boundary, and realizes flexible and automatic division of dangerous areas according to the changes in the scene background, thereby realizing personnel intrusion target detection and abnormal alarm, and is used for efficient and intelligent identification of personnel intrusion in dangerous areas of fully-mechanized working faces. The technical solution provided by the present invention can effectively solve the problems raised in the above-mentioned background technology, and provides certain technical support for the intelligent identification of fully-mechanized working faces and the safety of personnel. It is simple to operate and has certain promotion potential.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for intelligently identifying dangerous areas where personnel intrude into a fully-mechanized coal mining face, comprising the following steps:

[0006] Step S1, video image data collection of personnel, side guards and coal baffles at the fully mechanized mining face: the video images of personnel, side guards and coal baffles at the fully mechanized mining face in various states are collected by the camera arranged on the hydraulic support, and the collected video images are transmitted to the image processor; the image processor simulates the images of personnel in different environmental states of the mine by image noise addition and HSV channel change methods, and constructs a multi-feature sample image data set;

[0007] Step S2, video image defogging and enhancement preprocessing: the image processor reads the real-time video stream of the fully mechanized mining face of the coal mine, uses a method based on boundary constraints and nonlinear context regularization to defog the image, and uses an adaptive multi-scale Retinex algorithm with color protection to enhance the image;

[0008] Step S3, establishing a personnel intrusion recognition model based on the improved YOLOv8s target detection method: annotating the video image data, dividing the label categories into three categories: personnel, side guards, and coal baffles, and dividing the video image data into a training set and a test set; based on the improved YOLOv8s target detection method, training the personnel data set to establish a fully mechanized mining face personnel recognition model; based on the YOLOv8s target detection method, training the side guards data set and the coal baffles data set to establish a dangerous area recognition model;

[0009] Step S4, automatic division of dangerous areas for human intrusion: Based on the trained dangerous area recognition model, the side guard plate and the coal retaining plate are identified, and key points are selected according to the set conditions. The key points of the side guard plate anchor frame and the coal retaining plate anchor frame are connected to the image boundary respectively, and four intersection points and image corner points are connected end to end to form a closed area, so as to realize automatic division of dangerous areas for human intrusion;

[0010] Step S5, identification of dangerous areas of human intrusion: Identify and detect human targets, automatically draw anchor frames, and take the coordinates of the center point of the anchor frame C (c x , c y ) as the key point, analyze the position coordinate relationship between point C and the drawn dangerous area, judge whether point C is within the drawn dangerous area, and perform personnel intrusion identification and judgment; generate an alarm signal based on the judgment result and output an abnormal alarm.

[0011] In the above-mentioned method for intelligently identifying personnel intrusion into dangerous areas of a fully-mechanized coal mining face, after step S3 and before step S4, the fully-mechanized coal mining face personnel identification model and the dangerous area identification model are also subjected to TensorRT acceleration processing.

[0012] In the above-mentioned intelligent identification method for personnel intrusion into dangerous areas of coal mine fully mechanized working faces, the specific implementation process of the improved YOLOv8s target detection method in step S3 is as follows:

[0013] Step 301, integrating the SE attention module and the RFAConv attention module, combining the channel feature information and spatial feature information extracted by the two, focusing on the area of ​​interest of personnel, paying attention to the channels useful for personnel detection, and comprehensively capturing the global and local feature information of personnel under the interference of complex coal mine background;

[0014] Step 302: before fusing the features of the p3, p4, and p5 layers, add a CCNet module to adaptively capture the long-distance context information on the cross paths;

[0015] Step 303: Use the Res2Net module to replace the C2f backbone module Bottleneck as the main gradient flow branch to form a new C2f_Res2block module;

[0016] Step 304: Sub-pixel convolution layer is used to replace the interpolation method in the ASFF module to obtain an improved ASFF module; the improved ASFF module is introduced into the detection head part of YOLOv8, and feature maps of different adjustment layers are fused according to the adaptive weight ratio.

[0017] In the above-mentioned intelligent identification method for personnel intrusion into dangerous areas of fully mechanized coal mining working faces, the Shortcut of the C2f_Res2block module in step 303 is set to False, and the output is directly passed through the 1×1 convolution layer.

[0018] In the above-mentioned method for intelligently identifying dangerous areas of personnel intrusion in a fully mechanized coal mining face, when selecting key points according to the set conditions in step S4, any appropriate point in a certain area of ​​the anchor frame corner point is selected according to the accurate range of the dangerous area division in the actual scene; the specific process of automatically dividing the dangerous area of ​​personnel intrusion in step S4 is:

[0019] Step 401, load the .engine and .dll model files after TensorRT acceleration processing, detect all the side guards and coal baffles in the frame image, draw the target anchor frame, and temporarily store the coordinate values ​​of the upper left point, upper right point, lower right point, and lower left point of each anchor frame of the side guard and the coordinate values ​​of the upper left corner and upper right corner of the coal baffle anchor frame in 6 different sequences to form 6 groups of coordinate value lists Lx1, Lx2, Lx3, Lx4, Lx5, and Lx6;

[0020] Step 402: Using the x-coordinate value of each anchor frame as a reference object, arrange the above six groups of coordinate values ​​from small to large to form six new lists b1, b2, b3, b4, b5, and b6, and take the first and last two of the six new lists, for a total of six maximum point coordinate values ​​p1-p12;

[0021] Step 403: according to the camera installation position, the captured image is divided into two general directions: obliquely upward and obliquely downward; connecting the maximum value point p7 (p7 x , p7 y ) and p8(p8 x , p8 y ), calculate the slope k = -(p8 y -p7 y ) / (p8 x -p7 x ), where p7 x is the horizontal coordinate of p7, p7 y is the ordinate of p7, p8 x is the horizontal coordinate of p8, p8 y is the ordinate of p8;

[0022] Step 404, according to the number of side guards and coal baffles detected in a single frame, the identification numbers of side guards and coal baffles in a single frame are divided into four categories: the number of side guards and coal baffles identified in a single frame is not less than 2, the number of side guards identified is not less than 2 and the number of coal baffles identified is 1, the number of side guards identified is 1 and the number of coal baffles identified is not less than 2, and the rest of the two types of target identification numbers. According to these four categories, key point selection rules are set;

[0023] When k>0, when the number of single-frame recognitions of the side guard plate and the coal retaining plate is greater than 2, the minimum and maximum horizontal coordinates x are selected to correspond to the 1 / 2 point on the bottom edge of the side guard plate anchor frame. Right 4 / 5 o'clock and the upper left corner point p9 and the upper left corner point p10 of the coal retaining plate anchor frame; connect the two points of the guard plate anchor frame and the two points of the coal retaining plate anchor frame to form straight lines l1 and l2, and l1 intersects the image boundary at points (m 00 , n 00 ), (m 01 , n 01 ) two points, l2 intersects with the image boundary at points (m 10 , n 10 ), (m 11 , n 11 ) two points;

[0024] When k < 0, when the number of single-frame recognitions of the guard plate and the coal baffle is greater than 2, take the coordinates of the lower left corner and the lower right corner of the guard plate, p3 and p8, and the straight line l1 connecting the two points intersects with the image boundary at points (m 00 , n00 ), (m 01 , n 01 ) two points; take the maximum coordinates of the upper right corner of the coal retaining plate p11, p12, and connect the two points to form a straight line l2 that intersects the image boundary at (m 10 , n 10 ), (m 11 , n 11 ) two points;

[0025] Step 405: If the lines l1 and l2 have no intersection in the image, the width and height of the image are w and h, (m0, n h ) is the lower left corner of the image, (m w , n h ) is the lower right corner of the image; when k>0, connect the ends in sequence (m 00 , n 00 ), (m 01 , n 01 ), (m 11 , n 11 ), (m 10 , n 10 ), (m0, n h ) five points form a closed polygon, which is the dangerous area of ​​intrusion of personnel in the fully mechanized mining face; when k < 0, connect the end to the end (m 01 , n 01 ), (m 00 , n 00 ), (m w , n h ), (m 10 , n 10 ), (m 11 , n 11 ) The five points form a closed polygon, which is the dangerous area for personnel intrusion in the fully mechanized mining face;

[0026] If the lines l1 and l2 have an intersection point (m 20 , n 20 ); when k>0, connect end to end in sequence (m 00 , n 00 ), (m 20 , n 20 ), (m 10 , n 10 ), (m0, n h ) The four points form a closed polygon; when k < 0, connect them end to end (m 20 , n 20 ), (m 00 , n 00 ), (m w , n h ), (m10 , n 10 )The four points form a closed polygon.

[0027] In the above-mentioned method for intelligently identifying dangerous areas where personnel have intruded into a fully-mechanized coal mining face, in the remaining cases of step 404, key points are selected using key point data of the previous frame image.

[0028] In the above-mentioned intelligent identification method for personnel intrusion danger zone of fully mechanized coal mining working face, the position coordinate relationship between the analysis point C and the drawn danger zone in step S5 is determined to determine whether point C is within the drawn danger zone. When personnel intrusion identification is performed, point C is used as the key point for determination, and the drawn danger zone is represented as zone A. The ray method is used to determine whether point C is within zone A by determining the parity of the number of intersections between the ray drawn from point C and the boundary of polygonal zone A. The specific process is as follows:

[0029] Step 501: With point C as the origin, draw a ray through the polygonal area A, and loop through each edge of the polygonal area. During each loop, calculate the horizontal coordinate x of the intersection of the ray and the current edge. The calculation formula is as follows:

[0030]

[0031] Among them, (A x[i] ,A y[i] ) and (A x[j] ,A y[j] ) are the starting and ending coordinates of the sides of polygon A, (C x ,C y ) is the coordinate of the judgment point C;

[0032] Step 502: According to the judgment condition (A y[i] >C y )≠(A y[j] >C y ) Determine the relative position of the ray and the edge. If the y coordinates of the two vertices are different from the y coordinate of point C, it means that the ray intersects the edge. According to condition C x ﹤x determines whether the horizontal coordinate of the point is less than the horizontal coordinate of the intersection point. If so, the number of intersection points is increased by one;

[0033] Step 503, determine the parity of the number of intersection points. If the number of intersection points is an odd number, it means that point C is inside area A, and it is determined that a person has invaded the dangerous area; if the number of intersection points is an even number, it means that point C is outside area A, and it is determined that a person has not invaded the dangerous area.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] 1. Aiming at the special environment of mines, the present invention constructs a multi-feature data set of guard plates, coal retaining plates, and mine personnel targets. The data set is based on target images in different scenes, lighting, angles, and states, and transforms and enhances the images, so that the method can effectively obtain multi-feature information about the target under different environmental influencing factors, solving the problems of insufficient existing mine personnel intrusion identification data and single data feature information, thereby improving the reliability and generalization of the model training results, adapting to the complex environment of mines, and making the target detection results more accurate.

[0036] 2. Aiming at the problem of difficulty in extracting features of multi-scale and small-target personnel and low recognition rate in the process of identifying dangerous areas of personnel intrusion in the fully-mechanized working face, the present invention proposes a method for identifying multi-scale and small-target personnel in underground coal mines by improving YOLOv8s. The method focuses on the personnel area by introducing RFAConv-SE and CCNet attention modules to improve the network's ability to capture important information of the image; C2f_Res2block is used to effectively expand the range of the receptive field and better extract multi-scale and small-target feature information; and the improved ASFF module is introduced to enhance the network's attention to multi-scale features and detail information of personnel. This method can accurately extract personnel targets while meeting the real-time requirements of the mine, solving the problem of poor representation of multi-scale personnel and small-target features.

[0037] 3. Aiming at the limitations of existing methods such as fixed division of dangerous areas, poor flexibility, and difficulty in adapting to changing scenes, the present invention proposes a method for automatically dividing dangerous areas for personnel intrusion based on the selection of key points for identifying side guards and coal baffles. Through specific target detection, the key points of the anchor frames for identifying side guards and coal baffles are extracted, and the point selection judgment conditions are designed. The dangerous areas are automatically and accurately divided according to the different backgrounds where the personnel targets are located, and the method can adapt to the situation where the height of the dangerous area changes when the camera jumps or rotates. The method has simple steps, reasonable design, high flexibility, and low investment cost.

[0038] 4. The present invention establishes a personnel intrusion identification model and proposes a personnel intrusion status identification method based on the ray method. The personnel intrusion status is determined autonomously through the pixel coordinate relationship between the center position of the personnel target (which can represent the center position of the human body) and the position of the dangerous area, which solves the high misjudgment rate problem of the existing identification method, and can efficiently and accurately identify the personnel intrusion status and perform sound and light alarm prompts. This method is accurate and reliable in operation, and realizes the semi-supervised detection of personnel intrusion in the comprehensive mining face, which is of great significance to the efficient and intelligent production of coal mines and is easy to promote and use.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1is a flow chart of the identification method of the present invention;

[0041] Figure 2 A camera arrangement diagram of an embodiment of the present invention;

[0042] Figure 3 Comparison between the original image and the result image of the image preprocessing of the present invention;

[0043] Figure 4 The coal mining tunnel is photographed from two different directions by the camera of the embodiment of the present invention;

[0044] FIG5( a ) is a diagram showing the improved YOLOv8s network structure of the present invention;

[0045] FIG5( b ) is a network structure diagram of the RFAConv-SE attention module of the present invention;

[0046] FIG5( c ) is a diagram showing the network structure of the improved ASFF module of the present invention;

[0047] Figure 6 It is a schematic diagram of the principle of the identification method of the present invention;

[0048] Figure 7 This is a diagram of the field test results of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] like Figure 1 As shown, the method for intelligently identifying dangerous areas where personnel intrude into a fully-mechanized coal mining face of the present invention comprises the following steps:

[0050] Step S1, video image data collection of personnel, side guards and coal baffles at the fully mechanized mining face: the video images of personnel, side guards and coal baffles at the fully mechanized mining face in various states are collected by the camera arranged on the hydraulic support, and the collected video images are transmitted to the image processor; the image processor simulates the images of personnel in different environmental states of the mine by image noise addition and HSV channel change methods, and constructs a multi-feature sample image data set;

[0051] In the specific implementation, due to the complex environment of coal mines, uneven lighting, fog and dust, etc., when constructing the data set of mine personnel, side guards, and coal baffles, video images of various states such as different scenes, angles, lighting, colors, and postures are collected. At the same time, geometric transformations such as flipping and translation are performed on insufficient samples to enhance the data to balance positive and negative samples;

[0052] It should be noted that during the coal mine production process, there are fewer cases of human intrusion and the amount of data that can be collected is limited. Therefore, in order to meet the requirements of deep learning for a large amount of training data, reduce the cost of data collection and data labeling, and improve the accuracy of the model in prediction, the collected data is enhanced to balance the positive and negative sample data and increase the sample base. The initial data and the enhanced data together constitute the final data set of personnel, side guards, and coal baffles.

[0053] Step S2, video image defogging and enhancement preprocessing: the image processor reads the real-time video stream of the fully mechanized mining face of the coal mine, uses a method based on boundary constraints and nonlinear context regularization to defog the image, and uses an adaptive multi-scale Retinex algorithm with color protection (autoMSRCR algorithm) to enhance the image;

[0054] It should be noted that the video images collected by the camera are easily affected by factors such as heavy dust, low illumination, and electromagnetic noise interference, resulting in poor image quality. To address the problem of coal mine image quality, image defogging and enhancement processing are required to make the edge feature information of the image clear;

[0055] In the embodiment of the present invention, the real-time image of the Hikvision camera is read by using the rtsp streaming method.

[0056] Step S3, establishing a personnel intrusion recognition model based on the improved YOLOv8s target detection method: annotating the video image data, dividing the label categories into three categories: personnel, side guards, and coal baffles, and dividing the video image data into a training set and a test set; based on the improved YOLOv8s target detection method, training the personnel data set to establish a fully mechanized mining face personnel recognition model; based on the YOLOv8s target detection method, training the side guards data set and the coal baffles data set to establish a dangerous area recognition model;

[0057] In the specific implementation, the video image data is divided into a training set and a test set in a ratio of 8:2;

[0058] Step S4, automatic division of dangerous areas for human intrusion: Based on the trained dangerous area recognition model, the side guard plate and the coal retaining plate are identified, and key points are selected according to the set conditions. The key points of the side guard plate anchor frame and the coal retaining plate anchor frame are connected to the image boundary respectively, and four intersection points and image corner points are connected end to end to form a closed area, so as to realize automatic division of dangerous areas for human intrusion;

[0059] Step S5, identification of dangerous areas of human intrusion: Identify and detect human targets, automatically draw anchor frames, and take the coordinates of the center point of the anchor frame C (c x , c y) as the key point, analyze the position coordinate relationship between point C and the drawn dangerous area, judge whether point C is within the drawn dangerous area, and perform personnel intrusion identification and judgment; generate an alarm signal based on the judgment result and output an abnormal alarm.

[0060] In this embodiment, after step S3 and before step S4, the personnel identification model and the dangerous area identification model of the fully mechanized mining face are also accelerated by TensorRT.

[0061] It should be noted that the adopted Tensor RT will generate an optimization decision and be used for the forward propagation of the network to transform and optimize the network: delete unused network layers to avoid unnecessary calculations; fuse multiple layers into a single layer, reduce the depth and width of the network, reduce repeated calls to the same function, and improve processing speed;

[0062] In this embodiment, the specific implementation process of the improved YOLOv8s target detection method in step S3 is:

[0063] Step 301, integrating the SE attention module and the RFAConv attention module, combining the channel feature information and spatial feature information extracted by the two, focusing on the area of ​​interest of personnel, paying attention to the channels useful for personnel detection, and comprehensively capturing the global and local feature information of personnel under the interference of complex coal mine background;

[0064] Step 302: before fusing the features of the p3, p4, and p5 layers, add a CCNet module to adaptively capture long-distance context information on cross paths; promote the interaction between features at different locations, and improve feature extraction of overlapping and small target persons;

[0065] Step 303: Use the Res2Net module to replace the C2f backbone module Bottleneck as the main gradient flow branch to form a new C2f_Res2block module;

[0066] In this embodiment, the Shortcut of the C2f_Res2block module in step 303 is set to False, and the output is directly output through the 1×1 convolution layer;

[0067] Step 304: Sub-pixel convolution layer is used to replace the interpolation method in the ASFF module to obtain an improved ASFF module; the improved ASFF module is introduced into the detection head part of YOLOv8, and feature maps of different adjustment layers are fused according to the adaptive weight ratio.

[0068] In the specific implementation, the network recognition model is based on the improved YOLOv8s and YOLOv8s network framework. No pre-trained weights are used in the establishment process. The SGD optimizer is used, the initial learning rate value is 0.01, the batch size is set to 32, and the maximum number of epochs for a single model is 100. The model establishment and reasoning are completed, and the mean intersection over union (MIOU), pixel classification accuracy (PA), and recognition accuracy are used as evaluation indicators of the model training effect to evaluate the model in order to achieve the best recognition effect.

[0069] MIoU is a standard metric for semantic segmentation models. It first calculates the IoU (intersection over union of the true label and the predicted label) for each class, and then averages the IoU of all categories. The intersection over union is the intersection of the predicted expected and actual areas divided by the union of the two. The MIoU calculation formula is as follows:

[0070]

[0071] Where: k is the number of categories. Usually there is a background class in semantic segmentation, so k+1 is used, p ii represents the number of pixels predicted to be of the i-th category, p ij represents the total number of pixels of the i-th true category that are mistakenly predicted as the j-th category, p ji is the total number of pixels of the jth true category correctly predicted as the i-th category, is the total number of pixels of the i-th category in the image, i.e. the marked area, is the total number of pixels predicted by the model in the image as the i-th category, that is, the predicted area;

[0072] PA is also a standard metric for semantic segmentation models, which is the ratio of the number of pixels predicted to be correctly classified to the total number of pixels in the image. The calculation formula is as follows:

[0073]

[0074] In this embodiment, when selecting key points according to the set conditions in step S4, any appropriate point within a certain area of ​​the anchor frame corner point is selected according to the accurate range of the dangerous area division in the actual scene; the specific process of automatically dividing the dangerous area for human intrusion in step S4 is:

[0075] Step 401, load the .engine and .dll model files after TensorRT acceleration processing, detect all the side guards and coal baffles in the frame image, draw the target anchor frame, and temporarily store the coordinate values ​​of the upper left point, upper right point, lower right point, and lower left point of each anchor frame of the side guard and the coordinate values ​​of the upper left corner and upper right corner of the coal baffle anchor frame in 6 different sequences to form 6 groups of coordinate value lists Lx1, Lx2, Lx3, Lx4, Lx5, and Lx6;

[0076] Step 402: Using the x-coordinate value of each anchor frame as a reference object, arrange the above six groups of coordinate values ​​from small to large to form six new lists b1, b2, b3, b4, b5, and b6, and take the first and last two of the six new lists, for a total of six maximum point coordinate values ​​p1-p12;

[0077] Step 403: according to the camera installation position, the captured image is divided into two general directions: obliquely upward and obliquely downward; connecting the maximum value point p7 (p7 x , p7 y ) and p8(p8 x , p8 y ), calculate the slope k = -(p8 y -p7 y ) / (p8 x -p7 x ), where p7 x is the horizontal coordinate of p7, p7 y is the ordinate of p7, p8 x is the horizontal coordinate of p8, p8 y is the ordinate of p8;

[0078] Step 404, according to the number of side guards and coal baffles detected in a single frame, the identification numbers of side guards and coal baffles in a single frame are divided into four categories: the number of side guards and coal baffles identified in a single frame is not less than 2, the number of side guards identified is not less than 2 and the number of coal baffles identified is 1, the number of side guards identified is 1 and the number of coal baffles identified is not less than 2, and the rest of the two types of target identification numbers. According to these four categories, key point selection rules are set;

[0079] When k>0, when the number of single-frame recognitions of the side guard plate and the coal retaining plate is greater than 2, the minimum and maximum horizontal coordinates x are selected to correspond to the 1 / 2 point on the bottom edge of the side guard plate anchor frame. Right 4 / 5 o'clock and the upper left corner point p9 and the upper left corner point p10 of the coal retaining plate anchor frame; connect the two points of the guard plate anchor frame and the two points of the coal retaining plate anchor frame to form straight lines l1 and l2, and l1 intersects the image boundary at points (m 00 , n 00 ), (m 01 , n 01 ) two points, l2 intersects with the image boundary at points (m 10 , n 10 ), (m 11 , n 11 ) two points;

[0080] When k < 0, when the number of single-frame recognitions of the guard plate and the coal baffle is greater than 2, take the coordinates of the lower left corner and the lower right corner of the guard plate, p3 and p8, and the straight line l1 connecting the two points intersects with the image boundary at points (m00 , n 00 ), (m 01 , n 01 ) two points; take the maximum coordinates of the upper right corner of the coal retaining plate p11, p12, and connect the two points to form a straight line l2 that intersects the image boundary at (m 10 , n 10 ), (m 11 , n 11 ) two points;

[0081] Step 405: If the lines l1 and l2 have no intersection in the image, the width and height of the image are w and h, (m0, n h ) is the lower left corner of the image, (m w , n h ) is the lower right corner of the image; when k>0, connect the ends in sequence (m 00 , n 00 ), (m 01 , n 01 ), (m 11 , n 11 ), (m 10 , n 10 ), (m0, n h ) five points form a closed polygon, which is the dangerous area of ​​intrusion of personnel in the fully mechanized mining face; when k < 0, connect the end to the end (m 01 , n 01 ), (m 00 , n 00 ), (m w , n h ), (m 10 , n 10 ), (m 11 , n 11 ) The five points form a closed polygon, which is the dangerous area for personnel intrusion in the fully mechanized mining face;

[0082] If the lines l1 and l2 have an intersection point (m 20 , n 20 ); when k>0, connect end to end in sequence (m 00 , n 00 ), (m 20 , n 20 ), (m 10 , n 10 ), (m0, n h ) The four points form a closed polygon; when k < 0, connect them end to end (m 20 , n 20 ), (m 00 , n 00 ), (m w , nh ), (m 10 , n 10 )The four points form a closed polygon.

[0083] In the specific implementation, in the remaining cases of step 404, since it is impossible to select key points based on the identified side guard plates and coal retaining plate targets to determine the direction of the tunnel, and considering that the tunnel does not change much before and after the video frame, the key points are selected using the key point data of the previous frame image.

[0084] In this embodiment, the position coordinate relationship between the analysis point C and the drawn dangerous area in step S5 is determined to determine whether point C is within the drawn dangerous area. When performing personnel intrusion identification and judgment, point C is used as the key point for judgment, and the drawn dangerous area is represented as area A. The ray method is used to determine whether point C is within area A by determining the parity of the number of intersections between the ray drawn from point C and the boundary of the polygonal area A. The specific process is as follows:

[0085] Step 501: With point C as the origin, draw a ray through the polygonal area A, and loop through each edge of the polygonal area. During each loop, calculate the horizontal coordinate x of the intersection of the ray and the current edge. The calculation formula is as follows:

[0086]

[0087] Among them, (A x[i] ,A y[i] ) and (A x[j] ,A y[j] ) are the starting and ending coordinates of the sides of polygon A, (C x ,C y ) is the coordinate of the judgment point C;

[0088] Step 502: According to the judgment condition (A y[i] >C y )≠(A y[j] >C y ) Determine the relative position of the ray and the edge. If the y coordinates of the two vertices are different from the y coordinate of point C, it means that the ray intersects the edge. According to condition C x ﹤x determines whether the horizontal coordinate of the point is less than the horizontal coordinate of the intersection point. If so, the number of intersection points is increased by one;

[0089] Step 503, determine the parity of the number of intersection points. If the number of intersection points is an odd number, it means that point C is inside area A, and it is determined that a person has invaded the dangerous area; if the number of intersection points is an even number, it means that point C is outside area A, and it is determined that a person has not invaded the dangerous area.

[0090] When the present invention is used, Figure 2As shown in the figure, considering the main scene arrangement of the coal mining face and the requirements for the best camera protection and good quality of the captured images, the camera is installed on the top plate of the hydraulic support in the coal mining face. The on-site video data is collected and some of the data is processed to produce a multi-feature data set of personnel, side guards, and coal baffles. The image data is annotated by the labelimg annotation tool, and the label names are person, warning, and a respectively; the label file is saved in VOC format, i.e., .xml file, and the VOC format is converted to YOLO format, i.e., .txt file during training; after the data set is divided into a training set and a test set at an 8:2 ratio, a personnel recognition model for the fully mechanized mining face is established based on the improved YOLOv8s, and a dangerous area recognition model is established based on YOLOv8s; due to the special environmental conditions of the mine, the network transmission is slow and there is a large delay. In order to solve the problem of slow recognition speed and minimize the time consumption in the model recognition process, the trained model file is used to generate the corresponding .wts file, Cmake and VS2019 are used to generate .exe and .dll files, and TensorRT is used to convert the trained .pt model into .engine file to achieve accelerated processing of the model.

[0091] The real-time camera image is read through rtsp streaming, and the reading channel is set to 1. Each frame of the read image is defogged based on boundary constraints and nonlinear context regularization, and enhanced by AutoMSRCR with color gain weighting to make the image clearer and the target edge information more obvious, such as Figure 3 The original image and the processed image are shown. It can be found that the processed image is clearer than the original image and the feature information is more obvious.

[0092] The network structure diagram of the improved YOLOv8s of the present invention is shown in Figure 5(a), the network structure diagram of the RFAConv-SE attention module of the present invention is shown in Figure 5(b), and the network structure diagram of the improved ASFF module of the present invention is shown in Figure 5(c); the key point selection rules for automatic area demarcation set by the present invention are shown in Table 1.

[0093] Table 1

[0094]

[0095]

[0096] Load the personnel identification model and the dangerous area identification model, automatically identify the coal guard and the side guard, select all the side guards and coal guard vertices in each frame of the video, and store their corresponding pixel coordinate information in a list, and use the horizontal axis as a reference to sort each sequence from small to large to obtain the maximum value point. Because this embodiment is installed according to the on-site camera, the display of the coal mining face tunnel screen is mainly divided into two directions, such as Figure 4As shown in Figure 1, the selection of the most suitable point in different directions is different. Therefore, the extreme points of the guard plate and the coal retaining plate are connected to determine the positive and negative slope k value, and the most suitable extreme point connection is determined according to the judgment result, and the points intersecting with the image boundary at points (m 00 , n 00 ), (m 01 , n 01 ), (m 11 , n 11 ), (m 10 , n 10 ), determine the final area vertices and connect them end to end to delineate the danger zone, such as Figure 6 The ray method is used to determine whether the personnel target is within the danger zone based on the coordinate relationship between the center point of the personnel anchor frame and the position of the designated danger zone. The actual on-site detection and judgment results of the fully mechanized mining face are shown in the figure. Figure 7 If the center point C is identified in area A, the personnel have invaded the dangerous area, and an audible and visual alarm is given, and the personnel anchor frame changes from green to red; otherwise, the personnel anchor frame is green and the detection is normal; it has good use effect, efficient recognition, accurate and reliable, and realizes the intelligent recognition and judgment of personnel intrusion into the dangerous area of ​​the fully mechanized mining face, effectively ensuring the life safety of underground workers.

[0097] In summary, in order to achieve accurate and intelligent identification of personnel in the comprehensive mining working face, the present invention proposes an intelligent identification method for personnel intrusion into the dangerous area of ​​the comprehensive mining working face of a coal mine based on the improved YOLOv8s target detection algorithm, and constructs a multi-feature data set of personnel, side guard plates and coal retaining plates to adapt to the complex environment of the comprehensive mining working face; establishes a personnel identification model and a dangerous area identification model to lay a model foundation for automatic identification and demarcation of dangerous areas; adopts a defogging algorithm based on boundary constraints and nonlinear context regularization and a color gain weighted AutoMSRCR algorithm to process fog and dust and blurred images to improve the quality of the image to be tested; automatically divides the dangerous area according to the selected key points and distinguishes the intrusion status of personnel, realizes automatic analysis and processing of feature information, and can automatically adjust the dangerous area drawing of the input image, solving the problem of misjudgment caused by the inability to adaptively adjust the fixed area when the camera jumps or rotates; ensures the real-time and accuracy of the identification of personnel intrusion into the dangerous area of ​​the comprehensive mining working face, which is of great significance to ensuring the safety of underground personnel, the efficiency of production and the intelligence of safety monitoring.

[0098] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A method for intelligently identifying dangerous areas where personnel invade a fully-mechanized coal mining face, characterized in that: The method comprises the following steps: Step S1, video image data collection of personnel, side guards and coal baffles at the fully mechanized mining face: the video images of personnel, side guards and coal baffles at the fully mechanized mining face in various states are collected by the camera arranged on the hydraulic support, and the collected video images are transmitted to the image processor; the image processor simulates the images of personnel in different environmental states of the mine by image noise addition and HSV channel change methods, and constructs a multi-feature sample image data set; Step S2, video image defogging and enhancement preprocessing: the image processor reads the real-time video stream of the fully mechanized mining face of the coal mine, uses a method based on boundary constraints and nonlinear context regularization to defog the image, and uses an adaptive multi-scale Retinex algorithm with color protection to enhance the image; Step S3, establishing a personnel intrusion recognition model based on the improved YOLOv8s target detection method: annotating the video image data, dividing the label categories into three categories: personnel, side guards, and coal baffles, and dividing the video image data into a training set and a test set; based on the improved YOLOv8s target detection method, training the personnel data set to establish a fully mechanized mining face personnel recognition model; based on the YOLOv8s target detection method, training the side guards data set and the coal baffles data set to establish a dangerous area recognition model; Step S4, automatic division of dangerous areas for human intrusion: Based on the trained dangerous area recognition model, the side guard plate and the coal retaining plate are identified, and key points are selected according to the set conditions. The key points of the side guard plate anchor frame and the coal retaining plate anchor frame are connected to the image boundary respectively, and four intersection points and image corner points are connected end to end to form a closed area, so as to realize automatic division of dangerous areas for human intrusion; Step S5, identification of dangerous areas of human intrusion: Identify and detect human targets, automatically draw anchor frames, and take the coordinates of the center point of the anchor frame C (c x , c y ) as the key point, analyze the position coordinate relationship between point C and the drawn dangerous area, judge whether point C is within the drawn dangerous area, and perform personnel intrusion identification and judgment; generate an alarm signal based on the judgment result and output an abnormal alarm.

2. According to claim 1, a method for intelligently identifying dangerous areas of personnel intrusion in a fully-mechanized coal mining face is characterized in that: After step S3 and before step S4, the personnel identification model and the dangerous area identification model of the fully mechanized mining face are also accelerated by TensorRT.

3. A method for intelligently identifying dangerous areas of personnel intrusion in a fully-mechanized coal mining face according to claim 1 or 2, characterized in that: The specific implementation process of the improved YOLOv8s target detection method described in step S3 is: Step 301, integrating the SE attention module and the RFAConv attention module, combining the channel feature information and spatial feature information extracted by the two, focusing on the area of ​​interest of personnel, paying attention to the channels useful for personnel detection, and comprehensively capturing the global and local feature information of personnel under the interference of complex coal mine background; Step 302: before fusing the features of the p3, p4, and p5 layers, add a CCNet module to adaptively capture the long-distance context information on the cross paths; Step 303: Use the Res2Net module to replace the C2f backbone module Bottleneck as the main gradient flow branch to form a new C2f_Res2block module; Step 304: Sub-pixel convolution layer is used to replace the interpolation method in the ASFF module to obtain an improved ASFF module; the improved ASFF module is introduced into the detection head part of YOLOv8, and feature maps of different adjustment layers are fused according to the adaptive weight ratio.

4. The method for intelligently identifying dangerous areas of personnel intrusion in a fully-mechanized coal mining face according to claim 3 is characterized in that: In step 303, the Shortcut of the C2f_Res2block module is set to False, and the output is directly passed through the 1×1 convolution layer.

5. The method for intelligently identifying dangerous areas of personnel intrusion in a fully-mechanized coal mining face according to claim 1 is characterized in that: When selecting key points according to the set conditions in step S4, any appropriate point in a certain area of ​​the anchor frame corner point is selected according to the accurate range of the dangerous area division in the actual scene; the specific process of automatic division of the dangerous area for human intrusion in step S4 is: Step 401, load the .engine and .dll model files after TensorRT acceleration processing, detect all the side guards and coal baffles in the frame image, draw the target anchor frame, and temporarily store the coordinate values ​​of the upper left point, upper right point, lower right point, and lower left point of each anchor frame of the side guard and the coordinate values ​​of the upper left corner and upper right corner of the coal baffle anchor frame in 6 different sequences to form 6 groups of coordinate value lists Lx1, Lx2, Lx3, Lx4, Lx5, and Lx6; Step 402: Using the x-coordinate value of each anchor frame as a reference object, arrange the above six groups of coordinate values ​​from small to large to form six new lists b1, b2, b3, b4, b5, and b6, and take the first and last two of the six new lists, for a total of six maximum point coordinate values ​​p1-p12; Step 403: according to the camera installation position, the captured image is divided into two general directions: obliquely upward and obliquely downward; connecting the maximum value point p7 (p7 x , p7 y ) and p8(p8 x , p8 y ), calculate the slope k = -(p8 y -p7 y ) / (p8 x -p7 x ), where p7 x is the horizontal coordinate of p7, p7 y is the ordinate of p7, p8 x is the horizontal coordinate of p8, p8 y is the ordinate of p8; Step 404, according to the number of side guards and coal baffles detected in a single frame, the identification numbers of side guards and coal baffles in a single frame are divided into four categories: the number of side guards and coal baffles identified in a single frame is not less than 2, the number of side guards identified is not less than 2 and the number of coal baffles identified is 1, the number of side guards identified is 1 and the number of coal baffles identified is not less than 2, and the rest of the two types of target identification numbers. According to these four categories, key point selection rules are set; When k>0, when the number of single-frame recognitions of the side guard plate and the coal retaining plate is greater than 2, the 1 / 2 point on the bottom edge of the side guard plate anchor frame corresponding to the minimum and maximum horizontal coordinates x is selected. Right 4 / 5 o'clock and the upper left corner point p9 and the upper left corner point p10 of the coal retaining plate anchor frame; connect the two points of the guard plate anchor frame and the two points of the coal retaining plate anchor frame to form straight lines l1 and l2, and l1 intersects the image boundary at points (m 00 , n 00 ), (m 01 , n 01 ) two points, l2 intersects with the image boundary at points (m 10 , n 10 ), (m 11 , n 11 ) two points; When k < 0, when the number of single-frame recognitions of the guard plate and the coal baffle is greater than 2, take the coordinates of the lower left corner and the lower right corner of the guard plate, p3 and p8, and the straight line l1 connecting the two points intersects with the image boundary at points (m 00 , n 00 ), (m 01 , n 01 ) two points; take the maximum coordinates of the upper right corner of the coal retaining plate p11, p12, and connect the two points to form a straight line l2 that intersects the image boundary at (m 10 , n 10 ), (m 11 , n 11 ) two points; Step 405: If the lines l1 and l2 have no intersection in the image, the width and height of the image are w and h, (m0, n h ) is the lower left corner of the image, (m w , n h ) is the lower right corner of the image; when k>0, connect the ends in sequence (m 00 , n 00 ), (m 01 , n 01 ), (m 11 , n 11 ), (m 10 , n 10 ), (m0, n h ) five points form a closed polygon, which is the dangerous area of ​​intrusion of personnel in the fully mechanized mining face; when k < 0, connect the end to the end (m 01 , n 01 ), (m 00 , n 00 ), (m w , n h ), (m 10 , n 10 ), (m 11 , n 11 ) The five points form a closed polygon, which is the dangerous area for personnel intrusion in the fully mechanized mining face; If the lines l1 and l2 have an intersection point (m 20 , n 20 ); when k>0, connect end to end in sequence (m 00 , n 00 ), (m 20 , n 20 ), (m 10 , n 10 ), (m0, n h ) The four points form a closed polygon; when k < 0, connect them end to end (m 20 , n 20 ), (m 00 , n 00 ), (m w , n h ), (m 10 , n 10 )The four points form a closed polygon.

6. The method for intelligently identifying dangerous areas of personnel intrusion in a fully-mechanized coal mining face according to claim 5 is characterized by: In the remaining cases of step 404, key points are selected using key point data of the previous frame image.

7. A method for intelligently identifying dangerous areas of personnel intrusion in a fully-mechanized coal mining face according to claim 1 or 2, characterized in that: In step S5, the position coordinate relationship between the analysis point C and the drawn dangerous area is determined to determine whether the point C is within the drawn dangerous area. When performing personnel intrusion identification and judgment, point C is used as the key point for judgment, and the drawn dangerous area is represented as area A. The ray method is used to determine whether the point C is within area A by judging the parity of the number of intersections between the ray drawn from point C and the boundary of the polygonal area A. The specific process is as follows: Step 501: With point C as the origin, draw a ray through the polygonal area A, and loop through each edge of the polygonal area. During each loop, calculate the horizontal coordinate x of the intersection of the ray and the current edge. The calculation formula is as follows: Among them, (A x[i] ,A y[i] ) and (A x[j] ,A y[j] ) are the starting and ending coordinates of the sides of polygon A, (C x ,C y ) is the coordinate of the judgment point C; Step 502: According to the judgment condition (A y[i] >C y )≠(A y[j] >C y ) Determine the relative position of the ray and the edge. If the y coordinates of the two vertices are different from the y coordinate of point C, it means that the ray intersects the edge. According to condition C x ﹤x determines whether the horizontal coordinate of the point is less than the horizontal coordinate of the intersection point. If so, the number of intersection points is increased by one; Step 503, determine the parity of the number of intersection points. If the number of intersection points is an odd number, it means that point C is inside area A, and it is determined that a person has invaded the dangerous area; if the number of intersection points is an even number, it means that point C is outside area A, and it is determined that a person has not invaded the dangerous area.

Images