An intelligent identification method for personnel intrusion into a dangerous area of a fully mechanized coal mining face

Abstract

The application discloses a kind of intelligent identification methods for personnel intrusion dangerous area of fully mechanized coal face, comprising the following steps: S1, fully mechanized coal face personnel, back support, coal baffle video image data acquisition;S2, video image defogging, enhancement preprocessing;S3, personnel intrusion identification model is established based on improved YOLOv8s target detection method;S4, personnel intrusion dangerous area automatic division;S5, personnel intrusion dangerous area identification.The application is aimed at the specific scene of coal mine to be detected, and the dangerous area is adaptively drawn through feature detection, realizes the automatic flexible division of dangerous area, has strong adaptability, can realize the accurate identification and advance precontrol of dangerous area personnel intrusion of fully mechanized coal face of coal mine, improves mine production efficiency and safety protection, promotes the intelligent construction of fully mechanized coal face, effectively guarantees the life safety of miner.

Description

Technical Field

[0001] This invention belongs to the field of coal mine safety technology, specifically relating to an intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces. Background Technology

[0002] With the introduction of technologies such as artificial intelligence, 5G communication, and big data, the production operations of fully mechanized coal mining faces are developing towards automation and intelligence, striving to achieve remote monitoring and intelligent decision-making and automated collaborative operation of various systems to ensure the safe and efficient operation of the fully mechanized mining face. In fully mechanized mining faces, the area between the coal retaining plate and the coal face is the working area of ​​the coal mining machine and the scraper conveyor; personnel accidentally entering this area can cause unavoidable accidents. Traditional identification methods have certain limitations. Therefore, researching intelligent identification of personnel in fully mechanized mining faces and solving personnel safety issues is urgently needed.

[0003] For hazardous areas in mines, traditional fencing has a low safety factor, and the fixed area results in poor flexibility and timeliness. While traditional video surveillance can intelligently acquire and store information, it cannot perform corresponding analysis and processing, leading to a high rate of missed detections due to reliance on manual observation. Currently, there is research on deep learning-based intrusion detection in coal mines. One approach involves directly deploying cameras in the target detection area to detect intruders; another involves drawing a warning line or area to trigger tripwire or boundary crossing alarms. However, these methods have fixed areas and low flexibility. When the background image captured by the camera changes due to movement or rotation, false alarms and misjudgments are likely to occur, resulting in poor adaptability to scene changes. Summary of the Invention

[0004] The technical problem this invention aims to solve is to address the shortcomings of the prior art by providing an intelligent identification method for personnel intrusion into dangerous areas of fully mechanized coal mining faces. This method improves detection accuracy by using image dehazing and enhancement processing to make target contour information clearer; storing the anchor frame coordinates of the detected targets and sorting them using a specific method; extracting the coordinates of key points of the detected targets; and connecting the key points to their intersection with the image boundary to flexibly and automatically divide dangerous areas according to changes in the scene background. This enables personnel intrusion target detection and anomaly alarms, allowing for efficient and intelligent identification of personnel intrusion into dangerous areas of fully mechanized coal mining faces. The technical solution provided by this invention effectively solves the problems mentioned in the background art, providing technical support for intelligent identification and personnel safety assurance in fully mechanized coal mining faces. It is simple to operate and has a certain degree of applicability.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for intelligent identification of personnel intrusion into dangerous areas in fully mechanized coal mining faces, comprising the following steps:

[0006] Step S1: Video image data acquisition of personnel, side guards, and coal retaining plates in the fully mechanized mining face: Video images of personnel, side guards, and coal retaining plates in various states are acquired by cameras arranged on the hydraulic supports, and the acquired video images are transmitted to the image processor; The image processor simulates images of personnel in different environmental states in the mine by adding noise to the images and changing the HSV channel, and constructs a multi-feature sample image dataset;

[0007] Step S2, Video Image Dehazing and Enhancement Preprocessing: The image processor reads the real-time video stream of the coal mine fully mechanized mining face, uses a method based on boundary constraints and nonlinear context regularization to perform dehazing on the image, and uses the adaptive multi-scale Retinex algorithm with color protection to enhance the image.

[0008] Step S3: Establish a personnel intrusion identification model based on the improved YOLOv8s target detection method: Label the video image data, classifying them into three categories: personnel, side guards, and coal retaining plates. Divide the video image data into training and testing sets. Based on the improved YOLOv8s target detection method, train the personnel dataset to establish a personnel identification model for the fully mechanized mining face. Based on the YOLOv8s target detection method, train the side guard dataset and coal retaining plate dataset to establish a hazardous area identification model.

[0009] Step S4: Automatic division of personnel intrusion danger zone: Based on the trained danger zone recognition model, the side guard plate and coal retaining plate are identified. Key points are selected according to the set conditions. The key points of the side guard plate anchor frame and the coal retaining plate anchor frame are connected to the image boundary. The four intersection points and the image corner points are connected end to end to form a closed area, thereby realizing the automatic division of personnel intrusion danger zone.

[0010] Step S5, Personnel Intrusion Hazard Area Identification: Identify and detect personnel targets, automatically draw anchor frames, and obtain the coordinates C(c) of the center point of the anchor frame. x c y As a key point, analyze the positional coordinate relationship between point C and the drawn danger zone to determine whether point C is within the drawn danger zone and perform personnel intrusion identification; generate alarm signals based on the judgment results and output abnormal alarms.

[0011] The above-mentioned intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces further incorporates TensorRT acceleration processing on the personnel identification model and dangerous area identification model after step S3 and before step S4.

[0012] The above-mentioned intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces, specifically the improved YOLOv8s target detection method described in step S3, is as follows:

[0013] Step 301: Integrate the SE attention module and the RFAConv attention module, combine the channel feature information and spatial feature information extracted by both, focus on the region of interest of personnel, pay attention to the channels that are useful for personnel detection, and comprehensively capture the global and local feature information of personnel under the interference of complex coal mine background.

[0014] Step 302: Before fusing the features of layers p3, p4, and p5, add the CCNet module to adaptively capture long-distance contextual information on the cross paths;

[0015] Step 303: Replace the C2f backbone module Bottleneck with the Res2Net module as the main gradient flow branch to form a new C2f_Res2block module;

[0016] Step 304: Replace the interpolation method in the ASFF module with a subpixel convolutional layer to obtain an improved ASFF module; introduce the improved ASFF module into the detection head of YOLOv8, and fuse the feature maps of different adjustment layers according to the adaptive weight ratio.

[0017] In the above-mentioned intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces, in step 303, the Shortcut property of the C2f_Res2block module is set to False, and the output is directly passed through a 1×1 convolutional layer.

[0018] The above-mentioned intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces, in step S4, when selecting key points according to set conditions, selects any appropriate point within a certain area of ​​the anchor frame corner point based on the accurate range of the dangerous area division in the actual scene; the specific process of automatic division of dangerous areas for personnel intrusion in step S4 is as follows:

[0019] Step 401: Load the .engine and .dll model files after TensorRT acceleration processing, detect all the side protection plates and coal retaining plates in the frame image, draw the target anchor frame, and temporarily store the coordinate values ​​of the four vertices of the top left, top right, bottom right and bottom left of each anchor frame of the side protection plate and the coordinate values ​​of the top left and top right corners of the anchor frame of the coal retaining plate in 6 different sequences to form 6 sets of coordinate value lists Lx1, Lx2, Lx3, Lx4, Lx5 and Lx6;

[0020] Step 402: Using the x-coordinate value of each anchor frame as a reference, arrange the above 6 sets of coordinate values ​​from smallest to largest to form 6 new lists b1, b2, b3, b4, b5, b6. Take the first and last two values ​​from each of the 6 new lists, for a total of 6 extreme point coordinate values ​​p1-p12.

[0021] Step 403: Based on the camera's installation location, divide the captured image into two approximate directions: diagonally upward and diagonally downward; connect the lowest point p7 at the bottom right corner of the side panel. x p7 y ) and p8(p8) x p8 y ), calculate the slope k = -(p8) y -p7 y ) / (p8 x -p7 x ); among them, p7 x Let p7 be the x-coordinate. y p7 is the ordinate, p8 is the ordinate. x Let p8 be the x-coordinate. y The ordinate of p8;

[0022] Step 404: Based on the number of side guards and coal retainers detected in a single frame, the detection counts are divided into four categories: the number of side guards and coal retainers detected in a single frame is not less than 2; the number of side guards detected is not less than 2 and the number of coal retainers detected is 1; the number of side guards detected is 1 and the number of coal retainers detected is not less than 2; and the remaining cases are not included in the previous categories. The key point selection rules are set according to these four categories.

[0023] When k > 0, and the number of single-frame recognitions for both the side guard plate and the coal retaining plate is greater than 2, the minimum and maximum x-coordinates correspond to the bottom half points of the side guard plate anchor frame, respectively. 4 / 5 points on the right And the upper left corner points p9 and p10 of the coal retaining plate anchor frame; connect the two points of the side protection plate anchor frame and the two points of the coal retaining plate anchor frame respectively to form straight lines l1 and l2, where l1 intersects the image boundary at point (m) respectively. 00 n 00 ), (m 01 n 01 At points (m) and (l2), l2 intersects the image boundary at points (m) and (l2). 10 n 10 ), (m 11 n 11 Two points;

[0024] When k < 0, and the number of single-frame recognitions for both the side protection plate and the coal retaining plate is greater than 2, the coordinates of the extreme points p3 and p8 at the lower left and lower right corners of the side protection plate are taken. The straight line l1 connecting the two points intersects the image boundary at point (m). 00 n00 ), (m 01 n 01 Two points; take the extreme coordinates p11 and p12 of the upper right corner of the coal retaining plate, and the straight line l2 connecting the two points intersects the image boundary at (m) points respectively. 10 n 10 ), (m 11 n 11 Two points;

[0025] Step 405: If lines l1 and l2 have no intersection point within the graph, let the width and height of the graph be w and h, respectively, and (m0, n...) h (m) represents the bottom left corner of the image. w n h () represents the bottom right corner of the image; when k > 0, connect the first and last points (m) sequentially. 00 n 00 ), (m 01 n 01 ), (m 11 n 11 ), (m 10 n 10 ), (m0, n h Five points form a closed polygon, which is the danger zone for personnel intrusion into the fully mechanized mining face; when k < 0, connect the first and last points (m) sequentially. 01 n 01 ), (m 00 n 00 ), (m w n h ), (m 10 n 10 ), (m 11 n 11 The five points form a closed polygon, which is the danger zone for personnel to invade the fully mechanized mining face.

[0026] If lines l1 and l2 intersect at a point (m) within the image, 20 n 20 When k > 0, connect the first and last (m) sequentially. 00 n 00 ), (m 20 n 20 ), (m 10 n 10 ), (m0, n h The four points form a closed polygon; when k < 0, connect the first and last points in sequence (m 20 n 20 ), (m 00 n 00 ), (m w n h ), (m10 n 10 The four points form a closed polygon.

[0027] In the above-mentioned intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces, for the remaining cases in step 404, the key points are selected using the key point data of the previous frame image.

[0028] The above-mentioned intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces, in step S5, analyzes the positional coordinate relationship between point C and the drawn dangerous area to determine whether point C is within the drawn dangerous area. When identifying personnel intrusion, point C is used as the key point for judgment. The drawn dangerous area is represented as region A. Using the ray method, the parity of the number of intersections between the ray emanating from point C and the boundary of polygonal region A is used to determine whether point C is inside region A. The specific process is as follows:

[0029] Step 501: Draw a ray from point C as the origin, passing through the polygonal region A. Iterate through each edge of the polygonal region. During each iteration, calculate the x-coordinate of the intersection point of the ray and the current edge. The calculation formula is as follows:

[0030]

[0031] Among them, (A) x[i] A y[i] ) and (A x[j] A y[j] (C) are the start and end coordinates of the sides of polygon A, respectively. x C y ) is the coordinate of point C;

[0032] Step 502: Based on the discrimination condition (A) y[i] >C y )≠(A y[j] >C y Determine the relative position of the ray and the edge. If the y-coordinates of the two vertices are different from the y-coordinate of point C, then the ray intersects the edge. Based on condition C... x <x determines whether the x-coordinate of a point is less than the x-coordinate of an intersection point; if so, the number of intersection points is incremented by one.

[0033] Step 503: Determine the parity of the number of intersection points. If the number of intersection points is odd, it means that point C is inside area A, and it is determined that the person has invaded the danger zone. If the number of intersection points is even, it means that point C is outside area A, and it is determined that the person has not invaded the danger zone.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] 1. This invention addresses the unique environment of mines by constructing a multi-feature dataset of targets such as side walls, coal retaining walls, and mine personnel. This dataset is based on target images under different scenes, lighting, angles, and states, and undergoes transformation and enhancement processing to effectively acquire multi-feature information about targets under different environmental influencing factors. This solves the problems of insufficient data and limited feature information in existing mine personnel intrusion identification methods, thereby improving the reliability and generalization of model training results, adapting to the complex environment of mines, and making target detection results more accurate.

[0036] 2. This invention addresses the difficulties in feature extraction and low recognition rates of multi-scale, small-target personnel during the identification of intruders in hazardous areas of fully mechanized mining faces. It proposes an improved YOLOv8s method for identifying multi-scale, small-target personnel in coal mines. This method improves the network's ability to capture important image information by introducing RFAConv-SE and CCNet attention modules to focus on the personnel region. C2f_Res2block effectively expands the receptive field, better extracting multi-scale, small-target feature information. An improved ASFF module is introduced to enhance the network's attention to multi-scale features and detailed information of personnel. This method can accurately extract personnel targets while meeting the real-time requirements of mine operations, solving the problem of poor feature representation of multi-scale personnel and small targets.

[0037] 3. This invention addresses the limitations of existing methods, such as fixed hazardous area division, poor flexibility, and difficulty in adapting to changing scenarios. It proposes a method for automatically dividing hazardous areas for personnel intrusion based on selecting key points for identification of side guards and coal retaining plates. Through specific target detection, key points of the anchor frames of the side guards and coal retaining plates are extracted. Point selection judgment conditions are designed, and hazardous areas are automatically and accurately divided according to different backgrounds of the personnel target. This method can adapt to changes in the height of hazardous areas due to camera movement or rotation. The method is simple, rationally designed, highly flexible, and has low investment costs.

[0038] 4. This invention establishes a personnel intrusion identification model and proposes a method for determining personnel intrusion status based on the ray-mapping method. It autonomously determines the intrusion status by analyzing the pixel coordinate relationship between the center position of the target person (which can represent the center position of the human body) and the location of the danger zone. This solves the problem of high false positive rates in existing methods and can efficiently and accurately identify personnel intrusion status and provide audible and visual alarm prompts. This method is operational, accurate, and reliable, realizing semi-supervised detection of personnel intrusion in fully mechanized mining faces. It is of great significance for efficient and intelligent production in coal mines and is easy to promote and use.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] Figure 1This is a flowchart of the identification method of the present invention;

[0041] Figure 2 This is a camera layout diagram according to an embodiment of the present invention;

[0042] Figure 3 This is a comparison between the original image and the result image after image preprocessing according to the present invention;

[0043] Figure 4 These are two different directions of a coal mining roadway captured by a camera in an embodiment of the present invention;

[0044] Figure 5(a) shows the improved YOLOv8s network structure of this invention;

[0045] Figure 5(b) is a network structure diagram of the RFAConv-SE attention module of the present invention;

[0046] Figure 5(c) is a network structure diagram of the improved ASFF module of the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the principle of the identification method of the present invention;

[0048] Figure 7 This is a diagram showing the field test results of an embodiment of the present invention. Detailed Implementation

[0049] like Figure 1 As shown, the intelligent identification method for personnel intrusion into dangerous areas in fully mechanized coal mining faces of the present invention includes the following steps:

[0050] Step S1: Video image data acquisition of personnel, side guards, and coal retaining plates in the fully mechanized mining face: Video images of personnel, side guards, and coal retaining plates in various states are acquired by cameras arranged on the hydraulic supports, and the acquired video images are transmitted to the image processor; The image processor simulates images of personnel in different environmental states in the mine by adding noise to the images and changing the HSV channel, and constructs a multi-feature sample image dataset;

[0051] In practice, due to the complex underground environment of coal mines, uneven lighting, fog and dust are common, when constructing the dataset of mine personnel, side protection plates, and coal retaining plates, video images under various conditions such as different scenes, angles, lighting, colors, and postures are collected. At the same time, data enhancement such as flipping and translation is performed on insufficient samples to balance positive and negative samples.

[0052] It should be noted that in the coal mine production process, there are relatively few cases of personnel intrusion, and the amount of data that can be collected is limited. Therefore, in order to meet the requirements of deep learning for a large amount of training data, reduce the cost of data collection and labeling, and improve the accuracy of the model in prediction, data augmentation processing is performed on the collected data to balance the positive and negative sample data and increase the sample base. The initial data and the augmented data together form the final dataset of personnel, side panels, and coal retaining plates.

[0053] Step S2, Video Image Dehazing and Enhancement Preprocessing: The image processor reads the real-time video stream of the coal mine fully mechanized mining face, performs dehazing processing on the image using a method based on boundary constraints and nonlinear context regularization, and enhances the image using the adaptive multi-scale Retinex algorithm with color protection (autoMSRCR algorithm).

[0054] It should be noted that video images captured by cameras are easily affected by factors such as heavy dust, low light, and electromagnetic noise interference, resulting in poor image quality. To address the image quality issues in coal mines, image dehazing and enhancement processing are required to make the edge features of the images more prominent.

[0055] In this embodiment of the invention, the RTSP streaming method is used to read the real-time video from the Hikvision camera.

[0056] Step S3: Establish a personnel intrusion identification model based on the improved YOLOv8s target detection method: Label the video image data, classifying them into three categories: personnel, side guards, and coal retaining plates. Divide the video image data into training and testing sets. Based on the improved YOLOv8s target detection method, train the personnel dataset to establish a personnel identification model for the fully mechanized mining face. Based on the YOLOv8s target detection method, train the side guard dataset and coal retaining plate dataset to establish a hazardous area identification model.

[0057] In practice, the video image data is divided into a training set and a test set in an 8:2 ratio;

[0058] Step S4: Automatic division of personnel intrusion danger zone: Based on the trained danger zone recognition model, the side guard plate and coal retaining plate are identified. Key points are selected according to the set conditions. The key points of the side guard plate anchor frame and the coal retaining plate anchor frame are connected to the image boundary. The four intersection points and the image corner points are connected end to end to form a closed area, thereby realizing the automatic division of personnel intrusion danger zone.

[0059] Step S5, Personnel Intrusion Hazard Area Identification: Identify and detect personnel targets, automatically draw anchor frames, and obtain the coordinates C(c) of the center point of the anchor frame. x c yAs a key point, analyze the positional coordinate relationship between point C and the drawn danger zone to determine whether point C is within the drawn danger zone and perform personnel intrusion identification; generate alarm signals based on the judgment results and output abnormal alarms.

[0060] In this embodiment, after step S3 and before step S4, TensorRT acceleration processing is performed on the personnel identification model and the hazardous area identification model of the fully mechanized mining face.

[0061] It should be noted that the Tensor RT used generates an optimization decision and is used for the forward propagation of the network to transform and optimize the network: unused network layers are removed to avoid unnecessary computation; multiple layers are merged into a single layer to reduce the depth and width of the network, reduce repeated calls to the same functions, and improve processing speed.

[0062] In this embodiment, the specific implementation process of the improved YOLOv8s target detection method in step S3 is as follows:

[0063] Step 301: Integrate the SE attention module and the RFAConv attention module, combine the channel feature information and spatial feature information extracted by both, focus on the region of interest of personnel, pay attention to the channels that are useful for personnel detection, and comprehensively capture the global and local feature information of personnel under the interference of complex coal mine background.

[0064] Step 302: Before fusing the features of layers p3, p4, and p5, add the CCNet module to adaptively capture long-distance contextual information on the intersecting paths; promote the interaction between features at different locations, and improve feature extraction for overlapping and small target personnel;

[0065] Step 303: Replace the C2f backbone module Bottleneck with the Res2Net module as the main gradient flow branch to form a new C2f_Res2block module;

[0066] In this embodiment, the Shortcut of the C2f_Res2block module in step 303 is set to False, and the output is directly passed through a 1×1 convolutional layer;

[0067] Step 304: Replace the interpolation method in the ASFF module with a subpixel convolutional layer to obtain an improved ASFF module; introduce the improved ASFF module into the detection head of YOLOv8, and fuse the feature maps of different adjustment layers according to the adaptive weight ratio.

[0068] In practice, the network recognition model is based on the improved YOLOv8s and YOLOv8s network framework. During the model building process, pre-trained weights are not used. Instead, the SGD optimizer is used with an initial learning rate of 0.01, a batch size of 32, and a maximum training epochs of 100 for a single model. The model is then built and inferred. The model is evaluated using the Mean Intersection over Union (MIOU), pixel classification accuracy (PA), and recognition precision as evaluation metrics to achieve the best recognition performance.

[0069] MIoU is a standard metric for semantic segmentation models. It first calculates the IoU (Intersection over Union) for each class, and then averages the IoU across all classes. The IoU is the intersection of the predicted and actual regions divided by their union. The MIoU calculation formula is as follows:

[0070]

[0071] In the formula: k is the number of categories. In semantic segmentation, there is usually a background class, so k+1 is used; p ii p represents the number of pixels predicted to be in the i-th category. ij p represents the total number of pixels that are incorrectly predicted as belonging to the j-th class when they belong to the i-th true class. ji It is the total number of pixels that correctly predict the j-th true class as the ith class. The total number of pixels of the i-th class in the image, i.e., the labeled area. The total number of pixels in the graph that the model predicts as belonging to the i-th category, i.e., the prediction region;

[0072] PA is also a standard metric for semantic segmentation models, which is the ratio of correctly predicted pixels to the total number of pixels in the image. The calculation formula is as follows:

[0073]

[0074] In this embodiment, when selecting key points according to set conditions in step S4, any appropriate point within a certain area of ​​the anchor frame corner is selected based on the accurate range of the actual scene's dangerous area division; the specific process of automatically dividing the personnel intrusion dangerous area in step S4 is as follows:

[0075] Step 401: Load the .engine and .dll model files after TensorRT acceleration processing, detect all the side protection plates and coal retaining plates in the frame image, draw the target anchor frame, and temporarily store the coordinate values ​​of the four vertices of the top left, top right, bottom right and bottom left of each anchor frame of the side protection plate and the coordinate values ​​of the top left and top right corners of the anchor frame of the coal retaining plate in 6 different sequences to form 6 sets of coordinate value lists Lx1, Lx2, Lx3, Lx4, Lx5 and Lx6;

[0076] Step 402: Using the x-coordinate value of each anchor frame as a reference, arrange the above 6 sets of coordinate values ​​from smallest to largest to form 6 new lists b1, b2, b3, b4, b5, b6. Take the first and last two values ​​from each of the 6 new lists, for a total of 6 extreme point coordinate values ​​p1-p12.

[0077] Step 403: Based on the camera's installation location, divide the captured image into two approximate directions: diagonally upward and diagonally downward; connect the lowest point p7 at the bottom right corner of the side panel. x p7 y ) and p8(p8) x p8 y ), calculate the slope k = -(p8) y -p7 y ) / (p8 x -p7 x ); among them, p7 x Let p7 be the x-coordinate. y p7 is the ordinate, p8 is the ordinate. x Let p8 be the x-coordinate. y The ordinate of p8;

[0078] Step 404: Based on the number of side guards and coal retainers detected in a single frame, the detection counts are divided into four categories: the number of side guards and coal retainers detected in a single frame is not less than 2; the number of side guards detected is not less than 2 and the number of coal retainers detected is 1; the number of side guards detected is 1 and the number of coal retainers detected is not less than 2; and the remaining cases are not included in the previous categories. The key point selection rules are set according to these four categories.

[0079] When k > 0, and the number of single-frame recognitions for both the side guard plate and the coal retaining plate is greater than 2, the minimum and maximum x-coordinates correspond to the bottom half points of the side guard plate anchor frame, respectively. 4 / 5 points on the right And the upper left corner points p9 and p10 of the coal retaining plate anchor frame; connect the two points of the side protection plate anchor frame and the two points of the coal retaining plate anchor frame respectively to form straight lines l1 and l2, where l1 intersects the image boundary at point (m) respectively. 00 n 00 ), (m 01 n 01 At points (m) and (l2), l2 intersects the image boundary at points (m) and (l2). 10 n 10 ), (m 11 n 11 Two points;

[0080] When k < 0, and the number of single-frame recognitions for both the side protection plate and the coal retaining plate is greater than 2, the coordinates of the extreme points p3 and p8 at the lower left and lower right corners of the side protection plate are taken. The straight line l1 connecting the two points intersects the image boundary at point (m).00 n 00 ), (m 01 n 01 Two points; take the extreme coordinates p11 and p12 of the upper right corner of the coal retaining plate, and the straight line l2 connecting the two points intersects the image boundary at (m) points respectively. 10 n 10 ), (m 11 n 11 Two points;

[0081] Step 405: If lines l1 and l2 have no intersection point within the graph, let the width and height of the graph be w and h, respectively, and (m0, n...) h (m) represents the bottom left corner of the image. w n h () represents the bottom right corner of the image; when k > 0, connect the first and last points (m) sequentially. 00 n 00 ), (m 01 n 01 ), (m 11 n 11 ), (m 10 n 10 ), (m0, n h Five points form a closed polygon, which is the danger zone for personnel intrusion into the fully mechanized mining face; when k < 0, connect the first and last points (m) sequentially. 01 n 01 ), (m 00 n 00 ), (m w n h ), (m 10 n 10 ), (m 11 n 11 The five points form a closed polygon, which is the danger zone for personnel to invade the fully mechanized mining face.

[0082] If lines l1 and l2 intersect at a point (m) within the image, 20 n 20 When k > 0, connect the first and last (m) sequentially. 00 n 00 ), (m 20 n 20 ), (m 10 n 10 ), (m0, n h The four points form a closed polygon; when k < 0, connect the first and last points in sequence (m 20 n 20 ), (m 00 n 00 ), (m w nh ), (m 10 n 10 The four points form a closed polygon.

[0083] In specific implementation, for the remaining cases in step 404, since it is impossible to determine the roadway direction based on the identified sidewalls and coal retaining plates to select key points, and considering that the roadway changes little before and after the video frame, the key points are selected using the key point data from the previous frame image.

[0084] In this embodiment, the positional coordinate relationship between analysis point C and the drawn danger zone described in step S5, and the determination of whether point C is within the drawn danger zone, are used as key points for personnel intrusion identification. The drawn danger zone is represented as region A. Using the ray method, the parity of the number of intersections between the ray emanating from point C and the boundary of polygonal region A is used to determine whether point C is inside region A. The specific process is as follows:

[0085] Step 501: Draw a ray from point C as the origin, passing through the polygonal region A. Iterate through each edge of the polygonal region. During each iteration, calculate the x-coordinate of the intersection point of the ray and the current edge. The calculation formula is as follows:

[0086]

[0087] Among them, (A) x[i] A y[i] ) and (A x[j] A y[j] (C) are the start and end coordinates of the sides of polygon A, respectively. x C y ) is the coordinate of point C;

[0088] Step 502: Based on the discrimination condition (A) y[i] >C y )≠(A y[j] >C y Determine the relative position of the ray and the edge. If the y-coordinates of the two vertices are different from the y-coordinate of point C, then the ray intersects the edge. Based on condition C... x <x determines whether the x-coordinate of a point is less than the x-coordinate of an intersection point; if so, the number of intersection points is incremented by one.

[0089] Step 503: Determine the parity of the number of intersection points. If the number of intersection points is odd, it means that point C is inside area A, and it is determined that the person has invaded the danger zone. If the number of intersection points is even, it means that point C is outside area A, and it is determined that the person has not invaded the danger zone.

[0090] When using this invention, as Figure 2As shown, considering the main scene arrangement of the coal mining face and the requirements for optimal camera protection and good image quality, the camera was installed on the top plate of the hydraulic support in the coal mining face. On-site video data was collected and some data was processed to create multi-feature datasets for personnel, side protection plates, and coal retaining plates. Image data were labeled using the labelimg annotation tool, with labels named "person," "warning," and "a." The label files were saved in VOC format (.xml files), and during training, the VOC format was converted to YOLO format (.txt files). After the dataset was divided into training and testing sets in an 8:2 ratio, a personnel recognition model for the fully mechanized mining face and a hazardous area recognition model were established based on improved YOLOv8s. Due to the special environmental conditions of the mine, network transmission is slow and there is significant latency. To solve the problem of slow recognition speed and minimize the time consumed in the model recognition process, the corresponding .wts files were generated using the trained model files, and .exe and .dll files were generated using CMake and VS2019. TensorRT was used to convert the trained .pt model into an .engine file to accelerate the model processing.

[0091] The camera's live feed is read via RTSP streaming, with one reading channel. Each frame is then enhanced using AutoMSRCR (Auto-MSRCR) with color gain weighting, based on boundary constraints and non-linear context regularization, to make the image clearer and the target edge information more prominent. Figure 3 The image shown is the original image and the processed image. It can be seen that the processed image is clearer and the feature information is more obvious than the original image.

[0092] The improved YOLOv8s network structure diagram of this invention is shown in Figure 5(a), the RFAConv-SE attention module network structure diagram of this invention is shown in Figure 5(b), and the improved ASFF module network structure diagram of this invention is shown in Figure 5(c). The automatic region delineation key point selection rules set by this invention are shown in Table 1.

[0093] Table 1

[0094]

[0095]

[0096] Load personnel identification models and hazardous area identification models, automatically identify coal retaining plates and side guards, select all vertices of side guards and coal retaining plates in each frame of video, and store their corresponding pixel coordinate information in a list. Using the horizontal axis as a reference, sort each sequence from smallest to largest to obtain the maximum and minimum points. Because this embodiment is based on the installation of on-site cameras, the display of the coal mining face roadway is mainly divided into two directions, such as... Figure 4As shown, the optimal point selection varies depending on the direction. Therefore, the extreme points of the side protection plate and the coal retaining plate are connected to determine the sign of the slope k. Based on the determination result, the most suitable extreme point is determined and connected, intersecting the image boundary at point (m). 00 n 00 ), (m 01 n 01 ), (m 11 n 11 ), (m 10 n 10 ), determine the final region vertices and connect them end to end in sequence, delineate the danger zone, such as Figure 6 As shown. Using the ray method, the location of a personnel target within the danger zone is determined based on the coordinate relationship between the center point of the personnel anchor frame and the designated danger zone. The actual on-site detection and judgment results at the fully mechanized mining face are as follows. Figure 7 As shown. If personnel are detected in area A at the center point C, an audible and visual alarm will be triggered, and the personnel anchor frame will change from green to red; otherwise, the personnel anchor frame will remain green, indicating normal detection. This method is effective, efficient, accurate, and reliable, enabling intelligent identification and judgment of personnel intrusion into dangerous areas of the fully mechanized mining face, effectively protecting the lives of underground workers.

[0097] In summary, to achieve accurate and intelligent identification of personnel in fully mechanized mining faces, this invention proposes an intelligent identification method for personnel intrusion into dangerous areas of fully mechanized mining faces based on an improved YOLOv8s target detection algorithm. It constructs a multi-feature dataset of personnel, sidewalls, and coal retaining walls to adapt to the complex environment of fully mechanized mining faces; establishes personnel identification models and dangerous area identification models, laying the foundation for automatic identification and delineation of dangerous areas; employs a dehazing algorithm based on boundary constraints and nonlinear context regularization, and an AutoMSRCR algorithm with color gain weighting to process foggy and blurred images, improving the quality of the images under test; automatically delineates dangerous areas based on selected key points and judges personnel intrusion status, realizing automatic analysis and processing of feature information; it can automatically adjust the dangerous area drawing of the input image, solving the problem of misjudgment caused by the inability of fixed areas to adapt and adjust when the camera is moving or rotating; ensuring the real-time and accurate identification of personnel intrusion into dangerous areas of fully mechanized mining faces, which is of great significance for ensuring the safety of underground personnel, the efficiency of production, and the intelligentization of safety monitoring.

[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for intelligent identification of personnel intrusion into dangerous areas in fully mechanized coal mining faces, characterized in that, The method includes the following steps: Step S1: Video image data acquisition of personnel, side guards, and coal retaining plates in the fully mechanized mining face: Video images of personnel, side guards, and coal retaining plates in various states are acquired by cameras arranged on the hydraulic supports, and the acquired video images are transmitted to the image processor; The image processor simulates images of personnel in different environmental states in the mine by adding noise to the images and changing the HSV channel, and constructs a multi-feature sample image dataset; Step S2, Video Image Dehazing and Enhancement Preprocessing: The image processor reads the real-time video stream of the coal mine fully mechanized mining face, uses a method based on boundary constraints and nonlinear context regularization to perform dehazing on the image, and uses the adaptive multi-scale Retinex algorithm with color protection to enhance the image. Step S3: Establish a personnel intrusion identification model based on the improved YOLOv8s target detection method: Label the video image data, classifying them into three categories: personnel, side guards, and coal retaining plates. Divide the video image data into training and testing sets. Based on the improved YOLOv8s target detection method, train the personnel dataset to establish a personnel identification model for the fully mechanized mining face. Based on the YOLOv8s target detection method, train the side guard dataset and coal retaining plate dataset to establish a hazardous area identification model. The specific implementation process of the improved YOLOv8s target detection method described in step S3 is as follows: Step 301: Integrate the SE attention module and the RFAConv attention module, combine the channel feature information and spatial feature information extracted by both, focus on the region of interest of personnel, pay attention to the channels that are useful for personnel detection, and comprehensively capture the global and local feature information of personnel under the interference of complex coal mine background. Step 302: Before fusing the features of layers p3, p4, and p5, add the CCNet module to adaptively capture long-distance contextual information on the cross paths; Step 303: Replace the C2f backbone module Bottleneck with the Res2Net module as the main gradient flow branch to form a new C2f_Res2block module; Step 304: Replace the interpolation method in the ASFF module with a subpixel convolutional layer to obtain an improved ASFF module; introduce the improved ASFF module into the detection head of YOLOv8, and fuse the feature maps of different adjustment layers according to the adaptive weight ratio; Step S4: Automatic division of personnel intrusion danger zone: Based on the trained danger zone recognition model, the side guard plate and coal retaining plate are identified. Key points are selected according to the set conditions. The key points of the side guard plate anchor frame and the coal retaining plate anchor frame are connected to the image boundary. The four intersection points and the image corner points are connected end to end to form a closed area, thereby realizing the automatic division of personnel intrusion danger zone. Step S5, Personnel Intrusion Hazard Area Identification: Identify and detect personnel targets, automatically draw anchor frames, and obtain the coordinates C(c) of the center point of the anchor frame. x c y As a key point, analyze the positional coordinate relationship between point C and the drawn danger zone to determine whether point C is within the drawn danger zone and perform personnel intrusion identification; generate alarm signals based on the judgment results and output abnormal alarms.

2. The intelligent identification method for personnel intrusion into dangerous areas in a fully mechanized coal mining face according to claim 1, characterized in that: After step S3 and before step S4, TensorRT acceleration processing is applied to the personnel identification model and the hazardous area identification model of the fully mechanized mining face.

3. The intelligent identification method for personnel intrusion into dangerous areas in a fully mechanized coal mining face according to claim 1, characterized in that: In step 303, the Shortcut of the C2f_Res2block module is set to False, and the output is directly passed through a 1×1 convolutional layer.

4. The intelligent identification method for personnel intrusion into dangerous areas in a fully mechanized coal mining face according to claim 1, characterized in that: In step S4, when selecting key points based on set conditions, any appropriate point within a certain area of ​​the anchor frame corner is selected according to the accurate range of the actual scene's dangerous area division. The specific process of automatically dividing the personnel intrusion dangerous area in step S4 is as follows: Step 401: Load the .engine and .dll model files after TensorRT acceleration processing, detect all the side protection plates and coal retaining plates in the frame image, draw the target anchor frame, and temporarily store the coordinate values ​​of the four vertices of the top left, top right, bottom right and bottom left of each anchor frame of the side protection plate and the coordinate values ​​of the top left and top right corners of the anchor frame of the coal retaining plate in 6 different sequences to form 6 sets of coordinate value lists Lx1, Lx2, Lx3, Lx4, Lx5 and Lx6; Step 402: Using the x-coordinate value of each anchor frame as a reference, arrange the above 6 sets of coordinate values ​​from smallest to largest to form 6 new lists b1, b2, b3, b4, b5, b6. Take the first and last two values ​​from each of the 6 new lists, for a total of 6 extreme point coordinate values ​​p1-p12. Step 403: Based on the camera's installation location, divide the captured image into two approximate directions: diagonally upward and diagonally downward; connect the lowest point p7 at the bottom right corner of the side panel. x p7 y ) and p8(p8) x p8 y ), calculate the slope k = -(p8) y -p7 y ) / (p8 x -p7 x ); among them, p7 x Let p7 be the x-coordinate. y p7 is the ordinate, p8 is the ordinate. x Let p8 be the x-coordinate. y The ordinate of p8; Step 404: Based on the number of side guards and coal retainers detected in a single frame, the detection counts are divided into four categories: the number of side guards and coal retainers detected in a single frame is not less than 2; the number of side guards detected is not less than 2 and the number of coal retainers detected is 1; the number of side guards detected is 1 and the number of coal retainers detected is not less than 2; and the remaining cases are not included in the previous categories. The key point selection rules are set according to these four categories. When k > 0, and the number of single-frame recognitions for both the side guard plate and the coal retaining plate is greater than 2, the minimum and maximum x-coordinates correspond to the bottom half points of the side guard plate anchor frame, respectively. 4 / 5 points on the right And the upper left corner points p9 and p10 of the coal retaining plate anchor frame; connect the two points of the side protection plate anchor frame and the two points of the coal retaining plate anchor frame respectively to form straight lines l1 and l2, where l1 intersects the image boundary at point (m) respectively. 00 n 00 ), (m 01 n 01 At points (m) and (l2), l2 intersects the image boundary at points (m) and (l2). 10 n 10 ), (m 11 n 11 Two points; When k < 0, and the number of single-frame recognitions for both the side protection plate and the coal retaining plate is greater than 2, the coordinates of the extreme points p3 and p8 at the lower left and lower right corners of the side protection plate are taken. The straight line l1 connecting the two points intersects the image boundary at point (m). 00 n 00 ), (m 01 n 01 Two points; take the extreme coordinates p11 and p12 of the upper right corner of the coal retaining plate, and the straight line l2 connecting the two points intersects the image boundary at (m) points respectively. 10 n 10 ), (m 11 n 11 Two points; Step 405: If lines l1 and l2 have no intersection point within the graph, let the width and height of the graph be w and h, respectively, and (m0, n...) h (m) represents the bottom left corner of the image. w n h () represents the bottom right corner of the image; when k > 0, connect the first and last points (m) sequentially. 00 n 00 ), (m 01 n 01 ), (m 11 n 11 ), (m 10 n 10 ), (m0, n h Five points form a closed polygon, which is the danger zone for personnel intrusion into the fully mechanized mining face; when k < 0, connect the first and last points (m) sequentially. 01 n 01 ), (m 00 n 00 ), (m w n h ), (m 10 n 10 ), (m 11 n 11 The five points form a closed polygon, which is the danger zone for personnel to invade the fully mechanized mining face. If lines l1 and l2 intersect at a point (m) within the image, 20 n 20 When k > 0, connect the first and last (m) sequentially. 00 n 00 ), (m 20 n 20 ), (m 10 n 10 ), (m0, n h The four points form a closed polygon; when k < 0, connect the first and last points in sequence (m 20 n 20 ), (m 00 n 00 ), (m w n h ), (m 10 n 10 The four points form a closed polygon.

5. The intelligent identification method for personnel intrusion into dangerous areas in a fully mechanized coal mining face according to claim 4, characterized in that: For the remaining cases in step 404, the key points are selected using the key point data from the previous frame image.

6. A method for intelligent identification of personnel intrusion into dangerous areas in a fully mechanized coal mining face according to claim 1 or 2, characterized in that: In step S5, the positional coordinate relationship between analysis point C and the drawn danger zone is analyzed to determine whether point C is within the drawn danger zone. When performing personnel intrusion identification, point C is used as the key point for judgment. The drawn danger zone is represented as region A. Using the ray method, the parity of the number of intersections between the ray emanating from point C and the boundary of polygon region A is used to determine whether point C is inside region A. The specific process is as follows: Step 501: Draw a ray from point C as the origin, passing through the polygonal region A. Iterate through each edge of the polygonal region. During each iteration, calculate the x-coordinate of the intersection point of the ray and the current edge. The calculation formula is as follows: Among them, (A) x[i] A y[i] ) and (A x[j] A y[j] (C) are the start and end coordinates of the sides of polygon A, respectively. x C y ) is the coordinate of point C; Step 502: Based on the discrimination condition (A) y[i] >C y )≠(A y[j] >C y Determine the relative position of the ray and the edge. If the y-coordinates of the two vertices are different from the y-coordinate of point C, then the ray intersects the edge. Based on condition C... x <x determines whether the x-coordinate of a point is less than the x-coordinate of an intersection point; if so, the number of intersection points is incremented by one. Step 503: Determine the parity of the number of intersection points. If the number of intersection points is odd, it means that point C is inside area A, and it is determined that the person has invaded the danger zone. If the number of intersection points is even, it means that point C is outside area A, and it is determined that the person has not invaded the danger zone.

Images