A method for early warning of illegal and dangerous behaviors of workers in fully mechanized mining face of coal mines

Abstract

The present invention discloses a method for early warning of violations and dangerous behaviors of workers in a fully-mechanized coal mining face, which comprises: S1, obtaining images of workers in a fully-mechanized coal mining face environment through a camera, and performing image preprocessing; S2, training a worker frame detection model based on the preprocessed images, and determining whether dangerous behaviors occur by judging whether the frame of the workers enters or approaches the ROI area of ​​a scraper; training a worker frame detection model based on the preprocessed images, and determining whether dangerous behaviors occur by judging whether the frame of the workers and the ROI area of ​​a cutting head are close to each other. The present invention provides a method for early warning of violations and dangerous behaviors of workers in a fully-mechanized coal mining face, and performs real-time image analysis after preprocessing the acquired images, and timely identifies and warns workers of abnormal behaviors such as falling, climbing over into a scraper, and approaching a cutting head.

Description

Technical Field

[0001] The invention relates to an early warning method for illegal and dangerous behaviors of operators at a fully mechanized mining face in a coal mine, and belongs to the field of intelligent video monitoring of coal mines. Background Art

[0002] my country is a large coal producer and consumer. With the improvement of the mechanization and intelligence of domestic coal mines, a relatively safe production environment is provided for front-line operators, and better self-rescue equipment is equipped to reduce the occurrence of certain dangerous events. The underground environment of the mine, especially the coal mine comprehensive mining face, is complex, damp and cold, the road is muddy, and the light is dim, so it is easy for operators to commit some dangerous and illegal behaviors. Because in the coal mine comprehensive mining face, the coal mine comprehensive mining face is supported by a large hydraulic support, and the cutting head of the coal mining machine moves on the scraper. There are large mechanical equipment and transmission equipment such as coal mining machines and conveyor belts around the operators, and these dangerous and illegal behaviors are very likely to cause personal injury accidents. For example, the operator approaches the cutting head of the coal mining machine that is running automatically, the operator who is inspecting on the hydraulic support suddenly falls, and the operator illegally climbs over the working scraper, all of which may cause serious safety accidents. In order to avoid such accidents, it is particularly important to install an intelligent video surveillance system on the comprehensive mining face and identify dangerous behaviors such as falling, climbing, and approaching through video streams.

[0003] At present, in traditional scenarios (non-underground operation scenarios), there are already a large number of existing technologies to identify operators' falls, climbing over fixed areas, approaching dangerous equipment, etc., but this fall behavior model cannot be directly applied to underground operation scenarios. Because the underground operation scene is dimly lit, the images collected by the video stream are low-light images, which need to be processed. However, new noise is easily introduced during the low-light image data processing, which brings greater uncertainty to the subsequent image recognition. In addition, the underground operation environment is complex and there are many obstructions. These dangerous and illegal behaviors are quite different from daily falls, climbing over, approaching and other dangers or violations, which brings difficulties to the identification work.

[0004] Therefore, it is necessary to propose a set of identification and early warning for dangerous and illegal behaviors under the comprehensive mining operation surface. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for early warning of violations and dangerous behaviors of workers in coal mine comprehensive mining working faces. Real-time image analysis is performed after preprocessing the acquired images, and timely identification and early warning are carried out for abnormal behaviors of workers such as falling, climbing over into a scraper, and approaching a cutting head.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A method for early warning of illegal and dangerous behaviors of workers in a fully mechanized coal mining face, comprising:

[0008] S1. Obtain images of workers in a fully mechanized coal mining working face environment through a camera and perform image preprocessing;

[0009] S2. Based on the preprocessed images, the operator border detection model is trained to determine whether dangerous behavior occurs by judging whether the operator's border enters or approaches the scraper ROI area;

[0010] Based on the pre-processed images, the operator border detection model is trained to determine whether dangerous behavior occurs by judging whether the operator border and the cutting head ROI area are close to each other;

[0011] Based on the preprocessed images, the key points of the human skeleton of the operator are obtained through the key point extraction method, and the state and sequence prediction of the key points of the human skeleton of the operator are performed to determine whether the operator has fallen, thereby determining whether dangerous behavior has occurred.

[0012] Furthermore, the image preprocessing in step S1 includes the following steps:

[0013] S11, adjusting the brightness of the low-light image acquired by the camera;

[0014] S12, using a non-blind denoising network to denoise the image after brightness adjustment, the non-blind denoising network comprising:

[0015] A noise estimation subnetwork, wherein the noise estimation subnetwork adopts a five-layer fully convolutional network with a convolution kernel of 3×3 and 32 channels, and uses relu as an activation function to convert the noisy observation image into an estimated noise level map;

[0016] The non-blind denoising sub-network uses a 16-layer U-Net structure and learns residual mapping using residual learning to obtain the final denoised image result.

[0017] Further, the non-blind denoising network is used to denoise the image after the brightness is adjusted in step S12, comprising the following steps:

[0018] S121, converting the RGB image into an HSV image (H, S, V represent hue, saturation, and brightness, respectively). The reason for converting to HSV here is to change the brightness of the low-light image by adjusting only the brightness V without changing the hue and saturation;

[0019] S122, obtaining the average brightness of local pixels in the HSV image;

[0020] S123, obtaining the stretching coefficient of the HSV image brightness through the local pixel average brightness;

[0021] S124, adjusting the brightness of the HSV image by using a stretching coefficient;

[0022] S125, restoring the image stretched in step S124 to an RGB image;

[0023] S126, performing five-layer full convolution inference on the RGB image of step S125;

[0024] S127. Use the U-Net model to infer the result of the five-layer full convolution in step S125 to obtain the final denoised image result.

[0025] Further, the training of the operator border detection model in step S2 includes the following steps:

[0026] According to the denoised image result in step S127, a worker border detection model is trained using a YOLO-V3 network structure to complete the detection of the worker border;

[0027] The operator border detection model is used to identify operators, scrapers and cutting heads. It is necessary to perform data annotation on the denoised image results in step S127. The data annotation categories include operators, scrapers and cutting heads, which are used to train the operator border detection model.

[0028] Further, in step S2, whether a dangerous behavior occurs is determined by judging whether the frame of the operator enters or approaches the scraper ROI area, including the following steps:

[0029] S211, based on the video stream interception image of the denoised image result in step S127, the scraper ROI area is delineated, the coal mine fully mechanized mining face is provided with a camera array to detect the entire fully mechanized mining area, each camera manages a scraper ROI area, and all scraper ROI areas complete the coverage of the scraper by overlapping. After the camera array is deployed, a number of markers are placed on the fully mechanized mining face of the coal mine, and the scraper ROI area is uniformly delineated by intercepting the image through the video stream, and the delineated scraper ROI area needs to be larger than the actual scraper area;

[0030] S212, judging whether the operator has entered the ROI area by judging whether the operator's frame has entered the scraper ROI area, judging whether the operator has climbed over or entered the scraper, and thus judging whether a dangerous behavior has occurred, using the lower 1 / 5 position (foot position) of the center line of the operator's frame obtained as the operator's coordinates, and judging whether the operator has climbed over or entered the scraper through the relationship between the coordinates and the scraper ROI area;

[0031] S213, optimize the determination process, eliminate the jitter of the operator's frame, use the data of continuous multi-frame state determination combined with the sequence determination method to determine whether the operator's frame enters or approaches the scraper ROI area, use the sliding window method to record the continuous N-frame state, and obtain the state sequence at time t as follows:

[0032] S t-N+1 , S t-N+2 , S t-N+3 , ..., S t ;

[0033] The sequence state at time t will be determined based on the weight of the state sequence. The closer to time t, the higher the weight.

[0034] Further, the acquisition of the real-time ROI area of ​​the cutting head in step S2 includes the following steps:

[0035] S221, obtaining the border of the cutting head, i.e., the cutting head bbox, through the operator border detection model recognition, and correcting the cutting head bbox according to the shape of the nearby scraper ROI area;

[0036] S222, using the longitudinal midline of the cutting head bbox as the upper base of the cutting head ROI area;

[0037] S223, construct a straight lower base parallel to the upper base at 1 / 4 of the height below the cutting head;

[0038] S224. Starting from the two points on the upper bottom, draw rays parallel to the two sides of the trapezoid of the scraper ROI area, intersecting with the lower bottom of the structure to form a new trapezoid as the cutting head ROI area.

[0039] Further, in step S2, by determining whether the operator frame and the cutting head ROI area are close to each other, the following steps are included:

[0040] S225, when the cutting head does not move, whether the operator is close to the ROI area is determined by judging whether the operator frame is close to the cutting head ROI area, and whether the operator is close to the cutting head, thereby determining whether a dangerous behavior occurs;

[0041] S226, when the cutting head moves, the obtained real-time ROI area is compared with the operator's frame to determine whether the operator has entered the moving cutting head ROI area, thereby determining whether a dangerous behavior has occurred;

[0042] S227, optimize the determination process, eliminate the jitter of the operator's frame, use the data of continuous multi-frame state determination combined with the sequence determination method to determine whether the operator's frame and the cutting head ROI area are close to each other, use the sliding window method to record the continuous N frame state, and obtain the state sequence at time t as follows:

[0043] S t-N+1 , S t-N+2 , S t-N+3 , ..., S t ;

[0044] The sequence state at time t will be determined based on the weight of the state sequence. The closer to time t, the higher the weight.

[0045] Further, in step S2, the key points of the human skeleton of the operator are obtained by a key point extraction method, and the state and sequence prediction of the key points of the human skeleton of the operator are performed to determine whether the operator has fallen, including the following steps:

[0046] S231, based on the video stream of the denoised image result in step S127, using the OpenPose model to detect the key points of the human skeleton, for the video stream of the denoised image result, detecting the key points of the human skeleton in each frame in the video stream, and outputting a sequence of skeleton key points after matching and connecting;

[0047] S232, taking the skeleton key point sequence as input, using the factorization machine model, outputting the status data of the operator, and judging whether the operator has fallen;

[0048] S233, performing sequence prediction on the status of the operator in step S232 to solve the problem of inaccurate judgment on the status of the operator.

[0049] Further, in step S231, the key points of the human skeleton in each frame of the video stream are detected, and a sequence of the key points of the skeleton is output after matching and connecting, which includes the following steps:

[0050] S2311, picture or video decoding;

[0051] S2312, image color space conversion, and data pre-processing of the image for a non-blind denoising network;

[0052] S2313, performing skeleton key point extraction network reasoning process;

[0053] S2314, parsing the network output, filtering out the detected skeleton key points, matching and connecting to the valid skeleton key points, the skeleton key points are 25 key points;

[0054] If no skeleton key points are detected at this stage, that is, no one is active around the coal mining machine, there is no need to enter step S232.

[0055] Further, obtaining the factorization machine model in step S232 includes the following steps:

[0056] S2321, sorting the key point data from the skeleton key point extraction network into a set sequence, then marking the falling action from the video stream time frame, and dividing the data into a training set and a test set according to KFOLD;

[0057] S2322, initializing the parameters of the factor decomposition machine model, training the factor decomposition machine model, and saving the factor decomposition machine model;

[0058] S2323, load the trained model, send the output of the key point detection network to the trained model, and output the single-frame status: single person falls or does not fall.

[0059] Further, in step S233, the sequence prediction of the state of the operator is performed, including the following steps:

[0060] S2331, obtaining data from the operator status judgment in step S232;

[0061] S2332, split the data in step S2331 and use the training set therein to train the sequence prediction model, forward calculate the output value of each neuron, reversely calculate the error of each neuron, calculate the weight gradient of each item according to the error, and store the model after the verification set indicators converge;

[0062] S2333. Send the output of the state prediction phase to the trained model, and the output is the result of whether the person falls.

[0063] By adopting the above-mentioned technical scheme, during automatic coal mining, the present invention obtains environmental images and worker images of the comprehensive mining face through a camera, adopts a new non-blind denoising method to preprocess image data, uses a pedestrian bounding box detection model to identify the worker to obtain the bounding box and compares it with the pre-defined dangerous equipment ROI area to determine whether there is any dangerous behavior of falling, climbing over into the scraper, or approaching the cutting head. A method combining skeleton key point recognition and sequence model is used to identify the dangerous behavior of falling of workers around the equipment, and the algorithm of the judgment process is optimized, thereby improving the judgment accuracy and ensuring that the judgment results are accurate and timely. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a principle block diagram of a method for early warning of illegal and dangerous behaviors of workers in a fully-mechanized coal mining face of the present invention;

[0065] Figure 2 is a flow chart of the non-blind denoising method of the present invention;

[0066] Figure 3 A network diagram for detecting the border of an operator according to the present invention;

[0067] Figure 4 A network diagram of OpenPose, a key point detection network of the present invention;

[0068] Figure 5 This is a flow chart of the determination from video stream to falling behavior of the present invention. DETAILED DESCRIPTION

[0069] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0070] like Figures 1 to 5 As shown, a method for early warning of illegal and dangerous behaviors of workers in a fully mechanized mining face of a coal mine comprises:

[0071] S1. Obtain images of workers in a fully mechanized coal mining working face environment through a camera and perform image preprocessing;

[0072] S2. Based on the preprocessed images, the operator border detection model is trained to determine whether dangerous behavior occurs by judging whether the operator's border enters or approaches the scraper ROI area;

[0073] Based on the pre-processed images, the operator border detection model is trained to determine whether dangerous behavior occurs by judging whether the operator border and the cutting head ROI area are close to each other;

[0074] Based on the preprocessed images, the key points of the human skeleton of the operator are obtained through the key point extraction method, and the state and sequence prediction of the key points of the human skeleton of the operator are performed to determine whether the operator has fallen, thereby determining whether dangerous behavior has occurred.

[0075] Among them, the image of the original video stream taken by the camera is a low-light image, which needs to be adjusted in brightness and denoised, so image preprocessing is required. The input of image preprocessing is the original video stream, and the output is a video stream with adjusted brightness and denoised. The working environment of the comprehensive mining face in an underground coal mine is dim, and the images collected through the video stream are low-light images. Simply adjusting the brightness of the dark area will inevitably amplify the noise and artifacts in the dark area. A non-blind denoising network is used here to denoise the image after adjusting the brightness of the low-light image. The image preprocessing described in step S1 includes the following steps:

[0076] S11, adjusting the brightness of the low-light image acquired by the camera;

[0077] S12, using a non-blind denoising network to denoise the image after brightness adjustment, the non-blind denoising network comprising:

[0078] The noise estimation subnetwork adopts a five-layer fully convolutional network with a convolution kernel of 3×3 and 32 channels, and uses relu as an activation function to convert the noisy observation image into an estimated noise level map.

[0079] The non-blind denoising sub-network uses a 16-layer U-Net structure and learns residual mapping using residual learning to obtain the final denoised image result.

[0080] The step S12 uses a non-blind denoising network to denoise the image after adjusting the brightness, including the following steps:

[0081] S121, converting the RGB image into an HSV image (H, S, V represent hue, saturation, and brightness, respectively). The reason for converting to HSV here is to change the brightness of the low-light image by adjusting only the brightness V without changing the hue and saturation;

[0082] S122, obtaining the average brightness of local pixels in the HSV image;

[0083] S123, obtaining the stretching coefficient of the HSV image brightness through the local pixel average brightness;

[0084] S124, adjusting the brightness of the HSV image by using a stretching coefficient;

[0085] S125, restoring the image stretched in step S124 to an RGB image;

[0086] S126, performing five-layer full convolution inference on the RGB image of step S125;

[0087] S127. Use the U-Net model to infer the result of the five-layer full convolution in step S125 to obtain the final denoised image result.

[0088] The training of the operator border detection model in step S2 includes the following steps:

[0089] According to the denoised image result in step S127, a worker border detection model is trained using a YOLO-V3 network structure to complete the detection of the worker border;

[0090] The operator border detection model is used to identify operators, scrapers and cutting heads. It is necessary to perform data annotation on the denoised image results in step S127. The data annotation categories include operators, scrapers and cutting heads, which are used to train the operator border detection model.

[0091] Wherein, in step S2, whether a dangerous behavior occurs is determined by judging whether the frame of the operator enters or approaches the scraper ROI area, including the following steps:

[0092] S211, based on the video stream interception image of the denoised image result in step S127, the scraper ROI area is delineated, the coal mine fully mechanized mining face is provided with a camera array to detect the entire fully mechanized mining area, each camera manages a scraper ROI area, and all scraper ROI areas complete the coverage of the scraper by overlapping. After the camera array is deployed, a number of markers are placed on the fully mechanized mining face of the coal mine, and the scraper ROI area is uniformly delineated by intercepting the image through the video stream. The delineated scraper ROI area needs to be larger than the actual scraper area, and a buffer margin is reserved;

[0093] S212, judging whether the operator has entered the ROI area by judging whether the operator's frame has entered the scraper ROI area, judging whether the operator has climbed over or entered the scraper, and thus judging whether a dangerous behavior has occurred, using the lower 1 / 5 position (foot position) of the center line of the operator's frame obtained as the operator's coordinates, and judging whether the operator has climbed over or entered the scraper through the relationship between the coordinates and the scraper ROI area;

[0094] S213. Optimizing the judgment process can improve the accuracy of the judgment. The operator's frame may shake. When this shaking occurs near the edge of the scraper ROI area, it may cause repeated out-of-bounds state jumps. In order to eliminate this jump and operator frame shaking, the data of continuous multi-frame state judgment is combined with the sequence judgment method to determine whether the operator's frame enters or approaches the scraper ROI area. The sliding window method is used to record the continuous N frame state. The state sequence obtained at time t is:

[0095] S t-N+1 , S t-N+2 , S t-N+3 , ..., S t ;

[0096] The sequence state at time t will be determined based on the weight of the state sequence. The closer to time t, the higher the weight.

[0097] Among them, the acquisition of the real-time ROI area of ​​the cutting head in step S2, the ROI area of ​​the cutting head changes according to the movement and position of the cutting head. Because the ROI of the cutting head moves and cannot be delineated in advance, it is necessary to obtain the cutting head border identified by the operator's border detection model after correction, including the following steps:

[0098] S221, obtaining the border of the cutting head, i.e., the cutting head bbox, through the operator border detection model recognition, and correcting the cutting head bbox according to the shape of the nearby scraper ROI area;

[0099] S222, using the longitudinal midline of the cutting head bbox as the upper base of the cutting head ROI area;

[0100] S223, construct a straight lower base parallel to the upper base at 1 / 4 of the height below the cutting head;

[0101] S224. Starting from the two points on the upper bottom, draw rays parallel to the two sides of the trapezoid of the scraper ROI area, intersecting with the lower bottom of the structure to form a new trapezoid as the cutting head ROI area.

[0102] Among them, in step S2, by judging whether the operator frame and the cutting head ROI area are approaching each other, approaching the cutting head can be divided into the operator approaching the cutting head and the cutting head moving toward the operator, including the following steps:

[0103] S225, when the cutting head does not move, whether the operator is close to the ROI area is determined by judging whether the operator frame is close to the cutting head ROI area, and whether the operator is close to the cutting head, thereby determining whether a dangerous behavior occurs;

[0104] S226, when the cutting head moves, the real-time ROI area is obtained and compared with the operator's border to determine whether the operator has entered the moving cutting head ROI area, thereby determining whether dangerous behavior has occurred; the cutting head is moving, and it is possible that the cutting head is close to the operator, causing an injury incident, and such incidents need to be determined in the same way. The real-time cutting head ROI area obtained by the real-time cutting head bbox through the above ROI acquisition process is compared with the real-time position of the operator to determine whether the operator's border has entered the moving cutting head ROI area;

[0105] S227, optimizing the determination process can improve the accuracy of the determination, eliminate the jitter of the operator's frame, use the data of continuous multi-frame state determination combined with the sequence determination method to determine whether the operator's frame and the cutting head ROI area are close to each other, use the sliding window method to record the continuous N frame state, and obtain the state sequence at time t as follows:

[0106] S t-N+1 , S t-N+2 , S t-N+3 , ..., S t ;

[0107] The sequence state at time t will be determined based on the weight of the state sequence. The closer to time t, the higher the weight.

[0108] Among them, in step S2, the key points of the human skeleton of the operator are obtained by the key point extraction method, and the state and sequence prediction of the key points of the human skeleton of the operator are performed to determine whether the operator has fallen, including the following steps:

[0109] S231, based on the video stream of the denoised image result in step S127, using the OpenPose model to detect the key points of the human skeleton, for the video stream of the denoised image result, detecting the key points of the human skeleton in each frame in the video stream, and outputting a sequence of skeleton key points after matching and connecting;

[0110] S232, taking the skeleton key point sequence as input, using the factorization machine model, outputting the status data of the operator, and judging whether the operator has fallen;

[0111] S233, performing sequence prediction on the status of the operator in step S232 to solve the problem of inaccurate judgment on the status of the operator.

[0112] Among them, in step S231, detecting the key points of the human skeleton in each frame in the video stream, matching and connecting, and outputting a sequence of the key points of the skeleton, comprises the following steps:

[0113] S2311, picture or video decoding;

[0114] S2312, image color space conversion, and data pre-processing of the image for a non-blind denoising network;

[0115] S2313, performing skeleton key point extraction network reasoning process;

[0116] S2314, parsing the network output, filtering out the detected skeleton key points, and matching and connecting them to the valid skeleton key points;

[0117] Skeleton key point detection OpenPose outputs 25 skeleton key points, mainly covering the head, neck, shoulders, elbows, hands, hips, knees, feet and other parts;

[0118] If no skeleton key points are detected at this stage, that is, no one is active around the coal mining machine, there is no need to enter step S232.

[0119] Wherein, obtaining the factor decomposition machine model in step S232 includes the following steps:

[0120] S2321, sorting the key point data from the skeleton key point extraction network into a set sequence, then marking the falling action from the video stream time frame, and dividing the data into a training set and a test set according to KFOLD;

[0121] S2322, initializing the parameters of the factor decomposition machine model, training the factor decomposition machine model, and saving the factor decomposition machine model;

[0122] The specific training process of the factor decomposition machine model is to input the factor decomposition machine model, wherein Adam is used as the optimization function, the learning rate is set and the learning rate is continuously adjusted as the network is trained, and the performance of the network is tested by the cross-validation method. Since the learning rate is a hyperparameter in neural network training, the optimal point may be missed when its initial value is too large. If the value is too small, the network convergence speed will be too slow. Therefore, the specific value of the learning rate needs to be determined according to the convergence of the network during the experiment. The learning rate in this embodiment is α=0.001.

[0123] S2323, load the trained model, send the output of the key point detection network to the trained model, and output the single-frame status: single person falls or does not fall.

[0124] The step S233 performs sequence prediction on the state of the operator, including the following steps:

[0125] S2331, obtaining data from the operator status judgment in step S232;

[0126] S2332, split the data in step S2331 and use the training set therein to train the sequence prediction model, forward calculate the output value of each neuron, reversely calculate the error of each neuron, calculate the weight gradient of each item according to the error, and store the model after the verification set indicators converge;

[0127] S2333. Send the output of the state prediction phase to the trained model, and the output is the result of whether the person falls.

[0128] The input of the state judgment stage in step S232 is the output of the human skeleton key point extraction stage in step S231 (and the human skeleton key point output set is not empty). The human skeleton key point output set may have the skeleton key points of several people. This embodiment takes the case of a single person falling as an example. Whether the operator falls in the state judgment stage is the state output, including: single person-falling, single person-not falling.

[0129] The factor decomposition machine model not only considers the impact of a single feature on the output state (i.e. the impact of a single skeletal key point on whether or not a person falls), but also considers the impact of second-order combined features on the target value (i.e. the cross-features of two skeletal key points, such as the quadratic term where the key point on the arm is very close to the head, which can be used as a very important feature). The factor decomposition machine model adds quadratic cross terms compared to the linear model, so it has better fitting ability. Compared with SVM, which can also handle quadratic terms, the training efficiency of the factor decomposition machine is better than that of SVM. The formula of the factor decomposition machine model is as follows:

[0130]

[0131] Finally, through all the above steps, it is possible to monitor whether the workers in the comprehensive mining face of the coal mine have any violations and dangerous behaviors, so as to reduce the incidence of coal mine production safety accidents.

[0132] The specific embodiments described above further illustrate the technical problems, technical solutions and beneficial effects solved by the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for early warning of illegal and dangerous behaviors of workers in fully-mechanized mining working faces in coal mines, characterized in that: It includes: S1. Obtain images of workers in a fully mechanized coal mining working face environment through a camera and perform image preprocessing; S2. Based on the preprocessed images, a worker border detection model is trained to determine whether a dangerous behavior occurs by judging whether the worker's border enters or approaches the scraper ROI area; Based on the pre-processed images, the operator border detection model is trained to determine whether dangerous behavior occurs by judging whether the operator border and the cutting head ROI area are close to each other; Based on the preprocessed images, the key points of the human skeleton of the operator are obtained by the key point extraction method, and the state and sequence prediction of the key points of the human skeleton of the operator are performed to determine whether the operator has fallen, thereby determining whether a dangerous behavior has occurred; The image preprocessing in step S1 includes the following steps: S11, adjusting the brightness of the low-light image acquired by the camera; S12, using a non-blind denoising network to denoise the image after brightness adjustment, the non-blind denoising network comprising: A noise estimation subnetwork, wherein the noise estimation subnetwork adopts a five-layer fully convolutional network with a convolution kernel of 3×3 and 32 channels, and uses relu as an activation function to convert the noisy observation image into an estimated noise level map; A non-blind denoising sub-network, wherein the non-blind denoising sub-network uses a 16-layer U-Net structure and learns residual mapping using a residual learning method to obtain a final denoised image result; The step S12 uses a non-blind denoising network to denoise the image after brightness adjustment, including the following steps: S121, converting the RGB image into an HSV image; S122, obtaining the average brightness of local pixels in the HSV image; S123, obtaining a stretching coefficient of the HSV image brightness through the average brightness of local pixels; S124, adjusting the brightness of the HSV image by using a stretching coefficient; S125, restoring the image stretched in step S124 to an RGB image; S126, performing five-layer full convolution inference on the RGB image of step S125; S127, using the U-Net model to infer the result of the five-layer full convolution in step S125 to obtain the final denoised image result; The training of the operator border detection model in step S2 includes the following steps: According to the denoised image result in step S127, a worker border detection model is trained using a YOLO-V3 network structure to complete the detection of the worker border; The operator frame detection model is used to identify the operator, scraper and cutting head. The denoised image result in step S127 needs to be annotated with data. The data annotation categories include operator, scraper and cutting head, which are used to train the operator frame detection model. In step S2, whether a dangerous behavior occurs is determined by judging whether the operator's frame enters or approaches the scraper ROI area, and the steps include: S211, based on the video stream interception image of the denoised image result in step S127, the scraper ROI area is delineated, the coal mine fully mechanized mining face is provided with a camera array to detect the entire fully mechanized mining area, each camera manages a scraper ROI area, and all scraper ROI areas complete the coverage of the scraper by overlapping. After the camera array is deployed, a number of markers are placed on the fully mechanized mining face of the coal mine, and the scraper ROI area is uniformly delineated by intercepting the image through the video stream, and the delineated scraper ROI area needs to be larger than the actual scraper area; S212, judging whether the operator has entered the ROI area by judging whether the operator's frame has entered the scraper ROI area, judging whether the operator has climbed over or entered the scraper, and thus judging whether a dangerous behavior has occurred, using the lower 1 / 5 position of the center line of the operator's frame as the operator's coordinates, and judging the operator's climbing over or entering the scraper through the relationship between the coordinates and the scraper ROI area; S213, optimize the determination process, eliminate the jitter of the operator's frame, use the data of continuous multi-frame state determination combined with the sequence determination method to determine whether the operator's frame enters or approaches the scraper ROI area, use the sliding window method to record the continuous N-frame state, and obtain the state sequence at time t as follows: S t-N+1 ,S t-N+2 ,S t-N+3 ,...,S t ; The sequence state at time t will be determined based on the weight of the state sequence. The closer to time t, the higher the weight.

2. A method for early warning of illegal and dangerous behaviors of workers in coal mine fully mechanized mining working faces according to claim 1, characterized in that: The acquisition of the real-time ROI area of ​​the cutting head in step S2 includes the following steps: S221, obtaining the border of the cutting head, i.e., the cutting head bbox, through the operator border detection model recognition, and correcting the cutting head bbox according to the shape of the nearby scraper ROI area; S222, using the longitudinal midline of the cutting head bbox as the upper base of the cutting head ROI area; S223, construct a straight lower base parallel to the upper base at 1 / 4 of the height below the cutting head; S224. Starting from the two points on the upper bottom, draw rays parallel to the two sides of the trapezoid of the scraper ROI area, intersecting with the lower bottom of the structure to form a new trapezoid as the cutting head ROI area.

3. A method for early warning of illegal and dangerous behaviors of workers in coal mine fully mechanized mining working faces according to claim 2, characterized in that: In step S2, it is determined whether the operator frame and the cutting head ROI area are close to each other, including the following steps: S225, when the cutting head does not move, whether the operator is close to the ROI area is determined by judging whether the operator frame is close to the cutting head ROI area, and whether the operator is close to the cutting head, thereby determining whether a dangerous behavior occurs; S226, when the cutting head moves, the obtained real-time ROI area is compared with the operator's frame to determine whether the operator has entered the moving cutting head ROI area, thereby determining whether a dangerous behavior has occurred; S227, optimize the determination process, eliminate the jitter of the operator's frame, use the data of continuous multi-frame state determination combined with the sequence determination method to determine whether the operator's frame and the cutting head ROI area are close to each other, use the sliding window method to record the continuous N frame state, and obtain the state sequence at time t as follows: S t-N+1 ,S t-N+2 ,S t-N+3 ,...,S t ; The sequence state at time t will be determined based on the weight of the state sequence. The closer to time t, the higher the weight.

4. A method for early warning of illegal and dangerous behaviors of workers in a fully mechanized coal mining face according to claim 1, characterized in that: In step S2, the key points of the human skeleton of the operator are obtained by a key point extraction method, and the state and sequence prediction of the key points of the human skeleton of the operator are performed to determine whether the operator has fallen, including the following steps: S231, based on the video stream of the denoised image result in step S127, using the OpenPose model to detect the key points of the human skeleton, for the video stream of the denoised image result, detecting the key points of the human skeleton in each frame in the video stream, and outputting a sequence of skeleton key points after matching and connecting; S232, taking the skeleton key point sequence as input, using the factorization machine model, outputting the status data of the operator, and judging whether the operator has fallen; S233, performing sequence prediction on the status of the operator in step S232 to solve the problem of inaccurate judgment on the status of the operator.

5. A method for early warning of illegal and dangerous behaviors of workers in fully mechanized mining working faces in a coal mine according to claim 4, characterized in that: In step S231, the key points of the human skeleton in each frame of the video stream are detected, and a sequence of the key points of the skeleton is output after matching and connecting, which includes the following steps: S2311, picture or video decoding; S2312, image color space conversion, and data pre-processing of the image for a non-blind denoising network; S2313, performing skeleton key point extraction network reasoning process; S2314, parsing the network output, filtering out the detected skeleton key points, matching and connecting to the valid skeleton key points, the skeleton key points are 25 key points; If no skeleton key points are detected at this stage, that is, no one is active around the coal mining machine, there is no need to enter step S232.

6. A method for early warning of illegal and dangerous behaviors of workers in fully mechanized mining working faces in a coal mine according to claim 5, characterized in that: Obtaining the factorization machine model in step S232 includes the following steps: S2321, sorting the key point data from the skeleton key point extraction network into a set sequence, annotating the falling action from the video stream time frame, and dividing the data into a training set and a test set according to KFOLD; S2322, initializing the parameters of the factor decomposition machine model, training the factor decomposition machine model, and saving the factor decomposition machine model; S2323, load the trained model, send the output of the key point detection network to the trained model, and output the single-frame status: single person falls or does not fall.

7. A method for early warning of illegal and dangerous behaviors of workers in fully mechanized mining working faces in a coal mine according to claim 6, characterized in that: In step S233, the sequence prediction of the state of the operator is performed, including the following steps: S2331, obtaining data from the operator status judgment in step S232; S2332, split the data in step S2331 and use the training set therein to train the sequence prediction model, forward calculate the output value of each neuron, reversely calculate the error of each neuron, calculate the weight gradient of each item according to the error, and store the model after the verification set indicators converge; S2333. Send the output of the state prediction phase to the trained model, and the output is the result of whether the person falls.

Images