A coal mine underground safety early warning method based on semantic recognition

Abstract

The present application relates to the technical field of image recognition, and particularly relates to a coal mine underground safety early warning method based on semantic recognition, comprising: collecting a thermal infrared image set; determining a suspicious personnel area by using difference information of adjacent two thermal infrared images; obtaining a reserved joint key point of the suspicious personnel according to position information of the suspicious personnel area, and further determining a recording camera; obtaining a visible light image by using the recording camera; obtaining a reference smoking parameter; obtaining an actual smoking parameter according to a distance between the reserved joint key points of the suspicious personnel in the visible light image; obtaining a key point gathering trend based on the reference smoking parameter and the actual smoking parameter; obtaining a motion periodicity by using distance change of the reserved joint key points of the suspicious personnel; determining a smoking probability according to the key point gathering trend and the motion periodicity; and performing a graded warning on the suspicious personnel based on the smoking probability. The present application can improve the accuracy of coal mine underground safety early warning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a coal mine underground safety warning method based on semantic recognition. BACKGROUND

[0002] When coal workers work in the coal mine, they often have some unsafe behaviors, such as smoking, not wearing a safety helmet, etc. These behaviors can cause safety accidents, so it is necessary to warn workers of unsafe behaviors to improve the safety of coal mine operations.

[0003] Existing problems: In the prior art, the safety warning of workers' unsafe behavior in the coal mine mainly relies on an AI video monitoring system. The system can effectively identify obvious target features or large behavior amplitudes such as not wearing a safety helmet and off-duty through a deep learning algorithm. However, the core features of smoking behavior are relatively subtle and easy to deliberately block. At the same time, the complex environment of low illumination and high dust in the mine can seriously degrade the imaging quality of the camera, making it difficult to clearly present the key identification target, and even completely invalidating the existing detection model that relies on clear images and obvious apparent features. SUMMARY

[0004] The present application provides a coal mine underground safety warning method based on semantic recognition to solve the existing problems.

[0005] The coal mine underground safety warning method based on semantic recognition of the present application adopts the following technical scheme:

[0006] An embodiment of the present application provides a coal mine underground safety warning method based on semantic recognition, which comprises the following steps:

[0007] Collecting a set of thermal infrared images in the coal mine; determining a heat source region and a suspicious person region corresponding to the heat source region using the difference information of adjacent two frames of thermal infrared images;

[0008] According to the position information of the suspicious person region, obtaining the reserved joint key points of the suspicious person; based on the position and number of the reserved joint key points of the suspicious person, determining the recording camera of the suspicious person; obtaining the visible light image of the suspicious person using the recording camera of the suspicious person;

[0009] According to the distance between the reserved joint key points of the normal person in the visible light image of the normal person, obtaining the reference smoking parameter; according to the distance between the reserved joint key points of the suspicious person in the visible light image of the suspicious person, obtaining the actual smoking parameter;

[0010] Based on the reference smoking parameter and the actual smoking parameter, obtaining the key point gathering trend of the suspicious person;

[0011] Obtaining the motion periodicity of the suspicious person by using the distance change of the reserved joint key points of the suspicious person in the visible light image of the suspicious person;

[0012] Determining the smoking probability of the suspicious person according to the key point gathering trend and the motion periodicity of the suspicious person;

[0013] Based on the smoking probability of the suspicious person, the suspicious person is graded and warned.

[0014] Further, the specific steps of determining the heat source region and the suspicious person region corresponding to the heat source region based on the difference information of the adjacent two frames of thermal infrared images include the following steps:

[0015] Judging whether the pixel values at the same positions of the adjacent two frames of thermal infrared images are different;

[0016] In the case that the pixel values at the same positions of the adjacent two frames of thermal infrared images are different, obtaining the frame difference image of the current adjacent two frames of thermal infrared images;

[0017] Threshold segmentation is performed on the frame difference image to obtain a binary image;

[0018] Morphological opening operation is performed on the binary image to obtain a denoising image;

[0019] Connected domain analysis is performed on the denoising image to obtain all connected domains;

[0020] The area and temperature of each connected domain are obtained, and the connected domain whose area is within a preset area threshold range and whose temperature is greater than a preset temperature threshold is determined as a heat source region;

[0021] Each connected domain outside the connected domain corresponding to the heat source region in the denoising image is determined as a candidate person region;

[0022] For each heat source region, the minimum Euclidean distance between the current heat source region and each candidate person region is calculated, and the candidate person region whose minimum Euclidean distance is less than a preset distance threshold is determined as the suspicious person region corresponding to the current heat source region.

[0023] Further, the specific steps of obtaining the reserved joint key points of the suspicious person according to the position information of the suspicious person region include the following steps:

[0024] According to the actual position information of the suspicious person region, all reserved cameras of the suspicious person are determined;

[0025] The suspicious person region image is obtained by using each reserved camera;

[0026] According to the coordinate information of the suspicious person in the suspicious person area image, the reserved joint key points of the suspicious person are obtained.

[0027] Further, the actual position information of the suspicious person area is determined to determine all the reserved cameras of the suspicious person, including the following specific steps:

[0028] For each heat source area corresponding to the suspicious person area, the actual ground coordinate point of the suspicious person area is obtained;

[0029] According to the actual ground coordinate point of the suspicious person area and the Euclidean distance of the installation position of each visible light camera, all the reserved cameras of the suspicious person are determined.

[0030] Further, the reserved joint key points of the suspicious person are obtained according to the coordinate information of the suspicious person in the suspicious person area image, including the following specific steps:

[0031] The coordinate point of the suspicious person in the suspicious person area image is determined as the suspicious person coordinate point;

[0032] Based on the suspicious person coordinate point, a lightweight model is used to detect the suspicious person area image to obtain the bounding box of the suspicious person;

[0033] The contour of the bounding box of the suspicious person is extracted to obtain the contour area of the suspicious person;

[0034] The contour area of the suspicious person is input into a human pose estimation model to obtain the joint key points and the confidence of the suspicious person, and the joint key points with a confidence less than a preset confidence threshold are removed to obtain the reserved joint key points of the suspicious person.

[0035] Further, the reserved joint key points of the suspicious person are obtained based on the position and number of the reserved joint key points of the suspicious person to determine the recording camera of the suspicious person, including the following specific steps:

[0036] According to the position of the reserved joint key points of the suspicious person, the reserved joint key points of the suspicious person are divided into first key points and second key points; wherein the first key points include the key points at the head joint and the key points at the hand joint, and the second key points include the key points at the leg joint;

[0037] The first key points are assigned a first weight, and the second key points are assigned a second weight; wherein the first weight is greater than the second weight;

[0038] The number of first key points and the number of second key points are obtained;

[0039] The product of the number of first key points and the first weight, and the product of the number of second key points and the second weight are summed to obtain the observation degree of the suspicious person area image.

[0040] The average of the observation degrees of all suspicious person area images corresponding to each reserved camera is determined as the observation degree of each reserved camera, and the reserved camera corresponding to the maximum observation degree is determined as the recording camera of the suspicious person.

[0041] Further, the specific steps of obtaining the reference smoking parameter according to the distance between the reserved joint key points of the normal person in the visible light image of the normal person include the following:

[0042] Obtain the contour area of the normal person in the visible light image of the normal person;

[0043] Calculate the ratio of the circumscribed rectangle area of the contour area of the normal person to the convex hull area formed by the reserved joint key points of the normal person, and determine the ratio as the shrinkage ratio of each visible light image of the normal person. The average of the shrinkage ratios of all visible light images of the normal person is determined as the trunk shrinkage ratio of the normal person;

[0044] Determine the ratio of the distance between the left and right elbow key points to the distance between the left and right shoulder key points in the reserved joint key points of the normal person as the elbow-shoulder ratio of each visible light image of the normal person. The average of the elbow-shoulder ratios of all visible light images of the normal person is determined as the trunk elbow-shoulder ratio of the normal person;

[0045] Determine the ratio of the distance between the left and right wrist key points to the distance between the left and right shoulder key points in the reserved joint key points of the normal person as the wrist-shoulder ratio of each visible light image of the normal person. The average of the wrist-shoulder ratios of all visible light images of the normal person is determined as the trunk wrist-shoulder ratio of the normal person;

[0046] Determine the average of the distance between the right wrist and the right shoulder key points and the distance between the left wrist and the left shoulder key points in the reserved joint key points of the normal person as the wrist-shoulder distance of each visible light image of the normal person. The average of the wrist-shoulder distances of all visible light images of the normal person is determined as the trunk wrist-shoulder distance of the normal person;

[0047] Determine the trunk shrinkage ratio, the trunk elbow-shoulder ratio, the trunk wrist-shoulder ratio, and the trunk wrist-shoulder distance of the normal person as the reference smoking parameter.

[0048] Further, the specific steps of obtaining the key point gathering trend of the suspicious person based on the reference smoking parameter and the actual smoking parameter include the following:

[0049] Determine the trunk shrinkage ratio, the trunk elbow-shoulder ratio, the trunk wrist-shoulder ratio, and the trunk wrist-shoulder distance of the suspicious person as the actual smoking parameter;

[0050] The ratio of each parameter in the actual smoking parameter to the corresponding parameter in the reference smoking parameter is obtained to obtain a ratio sequence of the actual smoking parameter and the reference smoking parameter;

[0051] The ratio of the number of elements less than 1 in the ratio sequence to the number of all elements is denoted as n, and the average of the elements less than 1 is denoted as j;

[0052] The product of n and the-jth power of the base of the natural logarithm is determined as the key point gathering trend of the suspicious person.

[0053] Further, the distance change of the reserved joint key points of the suspicious person in the visible light image of the suspicious person is used to obtain the motion periodicity of the suspicious person, and the specific steps include the following:

[0054] For the visible light image of the suspicious person in the continuous frame, the Euclidean distance between the right wrist key point and the nose key point in the reserved joint key points of the suspicious person is calculated, and a sequence formed by all the Euclidean distances is determined as a distance sequence;

[0055] The distance sequence is subjected to STL decomposition to obtain a periodic term;

[0056] The Fourier transform is performed on the periodic term to obtain a frequency spectrum corresponding to the periodic term;

[0057] The frequency corresponding to the maximum amplitude in the frequency spectrum is determined as a main frequency, and the ratio of the maximum amplitude corresponding to the main frequency to the sum of all other amplitudes is determined as the motion periodicity of the suspicious person.

[0058] Further, the smoking probability of the suspicious person is determined according to the key point gathering trend and the motion periodicity of the suspicious person, and the specific steps include the following:

[0059] The product of the key point gathering trend and the motion periodicity of the suspicious person is determined as the smoking probability of the suspicious person.

[0060] The technical scheme of the present application has the following advantages: the coal mine underground safety warning method based on semantic recognition is proposed in the embodiment of the present application, which analyzes the thermal infrared image, identifies the heat source and the suspicious person in the image, then analyzes the abnormal change characteristics of the human body key points in the visible light image of the suspicious person to determine the smoking probability of the suspicious person, and finally performs safety warning on the suspicious person according to the smoking probability. The present application can improve the accuracy of coal mine underground safety warning. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0062] Figure 1 The step flow chart of the coal mine underground safety early warning method based on semantic recognition of the present application;

[0063] Figure 2 The human joint key point schematic diagram. DETAILED DESCRIPTION

[0064] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the following will combine the drawings and the preferred embodiments to specifically describe the specific implementation, structure, features and effects of the coal mine underground safety early warning method based on semantic recognition according to the present application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0066] The following will specifically describe the specific scheme of the coal mine underground safety early warning method based on semantic recognition provided by the present application in combination with the drawings.

[0067] With the development of science and technology, the coal mine safety monitoring system has basically realized the visual full coverage of the underground operation scene, and can produce massive image data every day. However, the current system mainly relies on manual monitoring or simple motion detection, and it is difficult to automatically identify the deep safety semantic information contained in the image, such as the loosening risk represented by the abnormal displacement of the equipment and the illegal operation implied by the personnel action. There is a complex nonlinear correlation between these image change information and safety hazards, and traditional methods cannot effectively mine their early warning value, resulting in that a large amount of valuable visual information cannot be converted into safety control power. The meaning of the behavior representation of the staff is obtained through the semantic recognition information of the environment context in the video data, so as to build an intelligent analysis bridge from visual features to safety semantics, and realize the deep perception and intelligent early warning of the enterprise safety production state.

[0068] In the prior art, the pre-warning of unsafe behavior of workers in the mine mainly relies on an AI video monitoring system deployed in key areas. The system can effectively identify conditions with obvious target features or large behavior amplitude such as not wearing a safety helmet (by accurately detecting head-worn objects) and off-duty (by analyzing the posture and residence time of personnel) through a deep learning algorithm.

[0069] The existing safety pre-warning system in the mine is effective in identifying conditions with obvious target features or large behavior amplitude such as not wearing a safety helmet and off-duty; however, this method is seriously unsuitable for application in smoking detection in the mine because the core features of smoking behavior (such as a cigarette and an ignition action) are more subtle and can be intentionally concealed, and at the same time, the complex environment of low illumination and high dust in the mine can seriously impair the imaging quality of the camera, not only making it difficult to clearly present the key identification target, but also obscuring the important visual feature of smoke, thereby causing the precision of the existing detection model relying on clear images and obvious apparent features to sharply decrease or even completely fail.

[0070] The present application adopts a fusion perception scheme mainly using thermal imaging and supplemented by a visible light camera: first, the high penetration of the thermal imaging camera in a low illumination and high dust environment is used to reliably and in real time detect and locate the body temperature of personnel and abnormal hot spots (such as a lit cigarette end) around them; once the system algorithm identifies a suspicious high temperature target, the linked high-definition visible light camera will be automatically triggered to quickly and accurately focus on the area for more detailed image analysis of the hand posture and handheld objects, thereby realizing the final accurate judgment and evidence collection of the smoking behavior; then, through real-time sound and light alarm, remote push of alarm information and automatic archiving of the violation evidence chain, efficient handling of hidden violation behaviors is realized to promote the standardized management of employee behavior.

[0071] Please refer to Figure 1 which shows a step flowchart of a coal mine underground safety pre-warning method based on semantic recognition provided by an embodiment of the present application, and the method comprises the following steps:

[0072] Step S001: collect a set of thermal infrared images in the coal mine; use the difference information of adjacent two frames of thermal infrared images to determine a heat source region and a suspicious personnel region corresponding to the heat source region.

[0073] It should be noted that: in this embodiment, both thermal infrared images and visible light images are involved, and two types of sensors need to be laid out in advance.

[0074] Specifically, the embodiment adopts a dual-spectrum integrated holder ball machine of a thermal infrared sensor and a visible light camera. It is installed in key areas underground, such as a roadway rest point, a transportation transfer station, an equipment operation point and other areas where illegal behaviors are prone to occur. It adopts a top-mounted or wall-mounted mode to ensure an unobstructed field of view and cover the key monitoring area. The installation height is recommended to be 2.5-3.5 meters, the depression angle is 30°-45°, and the optimal viewing angle is used to capture the upper body and hand movements of a person. A cross-view angle is used to avoid a monitoring blind area, and at least one device is required to clearly capture the face and hand details of a person in a key area. The resolution of the thermal infrared sensor is ≥384×288, the thermal sensitivity is <50mK, and an automatic focusing is used. The resolution of the visible light camera is 1920×1080, and the frame rate is 25-30fps.

[0075] After the layout, the thermal infrared sensor is used to perform image acquisition to obtain a thermal infrared image set underground.

[0076] In the embodiment, whether the pixel values at the same positions of two adjacent thermal infrared images are different is determined. In the case that the pixel values at the same positions of two adjacent thermal infrared images are different, a frame difference image of the current two adjacent thermal infrared images is obtained. The frame difference image is subjected to threshold segmentation to obtain a binary image. The binary image is subjected to a morphological opening operation to obtain a denoising image. The denoising image is subjected to connected domain analysis to obtain all connected domains. The area and temperature of each connected domain are obtained. A connected domain with an area within a preset area threshold range and a temperature greater than a preset temperature threshold is determined as a heat source region. Each connected domain outside the connected domain corresponding to the heat source region in the denoising image is determined as a candidate personnel region. For each heat source region, the minimum Euclidean distance between the current heat source region and each candidate personnel region is calculated. A candidate personnel region with a minimum Euclidean distance less than a preset distance threshold is determined as a suspicious personnel region corresponding to the current heat source region.

[0077] It should be noted that the preset area threshold range and the preset temperature threshold are set according to specific conditions, which are not limited here.

[0078] It should be noted that for each thermal infrared sensor, the frame difference image of the current frame and the previous frame is obtained, that is, the difference information is obtained. For a person smoking, when a lighter or other heat source is used to light a cigarette, the heat source suddenly appears. Therefore, in the embodiment, the isolated heat source suddenly appearing is first detected through the frame difference image, specifically as follows.

[0079] The image at the current time is denoted as a current frame image, and the image of the previous frame is denoted as a previous frame image.

[0080] The absolute value of the difference value of the pixel values at the same positions of the current frame image and the previous frame image is calculated to obtain a frame difference image.

[0081] A binary image is obtained by otsu threshold segmentation on the frame difference image.

[0082] A morphological opening operation is performed on the binary image to eliminate small noise points and obtain a denoised image.

[0083] Connected component analysis is performed on the denoised image, and false targets are filtered according to the area and temperature characteristics of the connected components, and the real isolated hot spots are retained, which are recorded as heat source regions.

[0084] For example, the connected component with an area of 5-50 pixels and a temperature significantly higher than 50% of the ambient temperature is determined as a heat source region. The area and temperature here need to be set by the user in combination with the actual underground temperature of the coal mine and the resolution of the thermal infrared sensor.

[0085] After finding the heat source, regardless of whether it is an igniter or a lit cigarette, the worker closest to the heat source, i.e., the worker who may want to smoke secretly, is found, and further analysis is performed, specifically:

[0086] In the binary image, the connected component includes the heat source and the worker. After determining the heat source region in the connected component as described above, the remaining connected component can be considered as the worker, i.e., each remaining connected component is determined as a candidate personnel region.

[0087] For each heat source region, the minimum Euclidean distance from the coordinate point of the heat source region to each candidate personnel region (i.e., the minimum distance between the heat source region and all pixel points in the candidate personnel region) is calculated. If the minimum Euclidean distance is less than a predetermined distance threshold c, the candidate personnel region corresponding to the minimum Euclidean distance is recorded as the suspicious personnel region corresponding to the heat source region.

[0088] It should be noted that the predetermined distance threshold is set according to the specific situation, and is not specifically limited here.

[0089] For example, c is 2 pixel points, which can be set by the user according to the resolution of the thermal infrared sensor.

[0090] Step S002: Obtain the retained joint key points of the suspicious personnel according to the position information of the suspicious personnel region; determine the recording camera of the suspicious personnel based on the positions and numbers of the retained joint key points of the suspicious personnel; and obtain the visible light image of the suspicious personnel using the recording camera of the suspicious personnel.

[0091] It should be noted that if the suspicious personnel is in a back or side state, the camera cannot observe the hand smoking posture. For each suspicious personnel, the distance from the suspicious personnel to each visible light camera is obtained. From the several visible light cameras with shorter distances, the visible light camera that can detect more hand key points of the suspicious personnel is determined, and then the behavior of the suspicious personnel is monitored through this camera.

[0092] In this embodiment, according to the actual position information of the suspicious person area, all reserved cameras of the suspicious person are determined; the image of the suspicious person area is obtained by using each reserved camera; and the reserved joint key points of the suspicious person are obtained according to the coordinate information of the suspicious person in the suspicious person area image.

[0093] In this embodiment, for each suspicious person area corresponding to a heat source area, the actual ground coordinate point of the suspicious person area is obtained; and all reserved cameras of the suspicious person are determined according to the actual ground coordinate point of the suspicious person area and the Euclidean distance between the installation position of each visible light camera.

[0094] Specifically, for each suspicious person, the center pixel coordinate of the suspicious person area is obtained; the center pixel coordinate of each suspicious person area, the three-dimensional coordinate of the installation position of the corresponding thermal infrared sensor, and the pre-calibrated parameters (such as focal length, distortion coefficient, etc.) are taken as the input of the reverse projection to obtain the world coordinate of the suspicious person in the underground, that is, the actual ground position of the suspicious person, a three-dimensional coordinate point, and the three-dimensional coordinate point is taken as the actual ground coordinate point of the suspicious person area.

[0095] After obtaining the actual ground position of each suspicious person, the Euclidean distance between the three-dimensional coordinate point of each suspicious person and the installation position of each visible light camera is calculated, and the five cameras with the smallest Euclidean distance are reserved, which are recorded as reserved cameras.

[0096] In this embodiment, the coordinate point of the suspicious person in the suspicious person area image is determined as the suspicious person coordinate point; based on the suspicious person coordinate point, a lightweight model is used to detect the suspicious person area image to obtain the bounding box of the suspicious person; the contour extraction is performed on the bounding box of the suspicious person to obtain the contour area of the suspicious person; the contour area of the suspicious person is input into a human pose estimation model to obtain the joint key points and the confidence of the suspicious person, and the joint key points with a confidence less than a preset confidence threshold are removed to obtain the reserved joint key points of the suspicious person.

[0097] Specifically, for each suspicious person, the three-dimensional coordinate of the actual ground position of the suspicious person and the spatial configuration parameter of the corresponding reserved camera are taken as the input, and through the PTZ (Pan-Tilt-Zoom, pan-tilt-zoom) servo control and automatic focusing technology, the high-definition video stream that has accurately aligned the person and automatically adjusted the focal length to be clear is obtained, so as to realize the close-up monitoring of the target behavior.

[0098] The three-dimensional coordinates of the actual ground position of each suspicious person, the parameters (internal parameter coefficients, distortion coefficients, etc.) of each reserved camera are taken as the input of the projection transformation, and the coordinates of the suspicious person in the shooting image of the corresponding reserved camera are obtained, which is recorded as the suspicious person coordinate point. The shooting image of the reserved camera is the suspicious person region image.

[0099] For each suspicious person region image, the suspicious person coordinate point is used as the center to detect the boundary box of the suspicious person using the YOLOv5n lightweight model. In subsequent video frames, the boundary box is tracked using the KCF (Kernelized Correlation Filters, target tracking) algorithm, so that the subsequent frames no longer need to be repeatedly detected, which can greatly save computing power.

[0100] The foreground is segmented in the current boundary box using the GrabCut (interactive image segmentation) algorithm, the contour of the suspicious person is extracted, and the contour region, i.e., the image of the suspicious person, is obtained.

[0101] The contour region is taken as the input, and the 17 human body joint key points of the suspicious person and their confidence are obtained by using the lightweight 2D human body pose estimation model MoveNet. The coordinate points with a confidence less than 0.6 are removed, indicating that the corresponding joint key points in the image are difficult to capture, and the reserved joint key points are obtained. The reserved joint key points are numbered in the order of Figure 2 Each key point is assigned a corresponding serial number, Figure 2 as shown in the human body joint key point diagram. Figure 2 As shown, the joint key points are in the order of: 0-nose, 1-left eye, 2-right eye, 3-left ear, 4-right ear, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 9-left wrist, 10-right wrist, 11-left hip, 12-right hip, 13-left knee, 14-right knee, 15-left ankle, and 16-right ankle.

[0102] It should be noted that the preset confidence threshold is set according to specific circumstances, and is not limited specifically here.

[0103] In this embodiment, the reserved joint key points of the suspicious person are divided into first key points and second key points according to the positions of the reserved joint key points of the suspicious person; wherein the first key points include the key points at the joint of the head and the key points at the joint of the hand, and the second key points include the key points at the joint of the leg; the first key points are assigned a first weight, and the second key points are assigned a second weight; wherein the first weight is greater than the second weight; the number of the first key points and the number of the second key points are obtained; the product of the number of the first key points and the first weight, and the product of the number of the second key points and the second weight are summed to obtain the observation degree of the suspicious person area image; the average of the observation degrees of all suspicious person area images corresponding to each reserved camera is determined as the observation degree of each reserved camera, and the reserved camera corresponding to the maximum observation degree is determined as the recording camera of the suspicious person.

[0104] It should be noted that the first weight and the second weight are specifically set according to specific conditions, which are not specifically limited here.

[0105] Specifically, for each reserved camera, the first 10 suspicious person area images are obtained. The key point serial numbers of the suspicious person in each image and the number of serial numbers are obtained.

[0106] For a person smoking, the main actions are the hands and the head, that is, the hands continuously deliver cigarettes to the mouth, so when calculating which camera observes the smoking action of the suspicious person best through key points, the hand and head key points are given a larger weight. Since other key points will also participate in the calculation in subsequent calculations, here the key points other than the hand and head key points are not directly removed, but are given a larger weight.

[0107] In this embodiment, the hand and head key points (i.e. key points with serial numbers 0-10) are given a larger weight. That is, when screening the reserved camera, the greater the probability of the hand and head key points corresponding to the reserved camera, the more likely it is to be the recording camera.

[0108] The weight can be set according to actual mine height, camera installation height, probability of 0-10 key points being easily detected, and other factors.

[0109] Wherein, the key points with serial numbers 0-10 are the first key points, and the key points with serial numbers 11-16 are the second key points.

[0110] The weight of each key point (the second key point) of the leg is represented by x (the second weight), and the weight of the other key points (the first key points) is represented by 1.2x (the first weight), and the weight of each key point is obtained by calculating 6x+11*1.2x=1.

[0111] For each reserved camera, the number of head and hand key points and the number of leg key points in each suspicious person region image thereof are obtained, and then the observation degree of each suspicious person region image is obtained by a weighted summation method, and the average of the observation degrees of the 10 suspicious person region images is taken as the observation degree of the suspicious person by each reserved camera. The greater the observation degree, the higher the accuracy of the reserved camera in identifying the smoking behavior of the suspicious person.

[0112] The reserved camera corresponding to the maximum observation degree is taken as the recording camera of the suspicious person. The image taken by the recording camera of the suspicious person is recorded as the visible light image of the suspicious person.

[0113] At this point, the recording camera of the suspicious person is obtained, and then whether the suspicious person is a secretly smoking employee is further identified through the video of the recording camera.

[0114] Step S003: obtaining a reference smoking parameter according to the distance between the reserved joint key points of the normal person in the visible light image of the normal person, and obtaining an actual smoking parameter according to the distance between the reserved joint key points of the suspicious person in the visible light image of the suspicious person.

[0115] It should be noted that in the coal mine, smoking behavior is strictly prohibited. In order to evade supervision, the irregular personnel usually smokes in a very hidden way, for example, uses body posture, clothes or surrounding environment for deliberate shielding. This deliberate hidden behavior will directly reflect in the monitoring video as the spatial distribution of the human joint key points showing significant abnormal changes. Compared with the relaxed posture when standing or walking normally, the smoker will unconsciously hunch back, shrug shoulders, bend elbows, or keep arms in a narrow space in front of the chest for a long time to shield the smoke and cigarettes. Therefore, the overall distribution of the human key points will show a clear tendency of inward contraction, curling, or the spatial feature that the key points (such as double wrists and double elbows) are highly concentrated in the middle line area of the body.

[0116] In the visible light image of the normal personnel obtained in this embodiment, the contour area of the normal personnel is obtained; the ratio of the area of the circumscribed rectangle of the contour area of the normal personnel to the area of the convex hull formed by the reserved joint key points of the normal personnel is calculated, and the ratio is determined as the shrinkage ratio of the visible light image of each normal personnel; the mean value of the shrinkage ratios of the visible light images of all normal personnel is determined as the trunk shrinkage ratio of the normal personnel; the ratio of the distance between the left and right elbow key points to the distance between the left and right shoulder key points in the reserved joint key points of the normal personnel is determined as the elbow-shoulder ratio of the visible light image of each normal personnel; the mean value of the elbow-shoulder ratios of the visible light images of all normal personnel is determined as the trunk elbow-shoulder ratio of the normal personnel; the ratio of the distance between the left and right wrist key points to the distance between the left and right shoulder key points in the reserved joint key points of the normal personnel is determined as the wrist-shoulder ratio of the visible light image of each normal personnel; the mean value of the wrist-shoulder ratios of the visible light images of all normal personnel is determined as the trunk wrist-shoulder ratio of the normal personnel; the mean value of the distance between the right wrist and the right shoulder key point and the distance between the left wrist and the left shoulder key point in the reserved joint key points of the normal personnel is determined as the wrist-shoulder distance of the visible light image of each normal personnel; the mean value of the wrist-shoulder distances of the visible light images of all normal personnel is determined as the trunk wrist-shoulder distance of the normal personnel; and the trunk shrinkage ratio, the trunk elbow-shoulder ratio, the trunk wrist-shoulder ratio, and the trunk wrist-shoulder distance of the normal personnel are determined as the reference smoking parameters.

[0117] Specifically, a large amount of key point data of the behaviors of the downhole personnel in a normal state, such as walking, standing and the like, is collected, and the key point data with a key point number greater than 14 is reserved, and specifically:

[0118] The visible light image of the normal personnel is obtained; the bounding box of the normal personnel is obtained by detecting the visible light image of the normal personnel; the contour area of the normal personnel is obtained by contour extraction on the bounding box of the normal personnel; the joint key points and the confidence of the normal personnel are obtained by inputting the contour area of the normal personnel into a human pose estimation model; the joint key points with a confidence less than a preset confidence threshold are removed, and the reserved joint key points of the normal personnel are obtained. The visible light image with more than 14 reserved joint key points is reserved for subsequent calculation.

[0119] When the personnel secretly shields the smoking behavior, the body will involuntarily curl up to avoid being found, and thus the trunk shrinkage ratio increases.

[0120] For each reserved visible light image, the ratio of the area of the circumscribed rectangle of the normal personnel contour area to the area of the convex hull formed by the reserved joint key points is calculated as the shrinkage ratio of the reserved visible light image. The mean value of the shrinkage ratios of all reserved visible light images is calculated as the trunk shrinkage ratio of the downhole personnel in a normal state, denoted as a1; that is, the trunk shrinkage ratio of the normal personnel is obtained by the above method.

[0121] When the person secretly blocks the smoking behavior, the body is curled inward, and the distance between key points 7 and 8 and the distance between key points 5 and 6 are reduced, and the ratio of the distance between key points 9 and 10 and the distance between key points 5 and 6 is also reduced.

[0122] For each reserved visible light image, the ratio of the Euclidean distance between key points 7 and 8 and the Euclidean distance between key points 5 and 6 is calculated as the elbow-shoulder ratio of the reserved visible light image. The average of the elbow-shoulder ratios of all reserved visible light images is calculated and denoted as a2; that is, the trunk elbow-shoulder ratio of the normal person is obtained through the above method.

[0123] For each reserved visible light image, the ratio of the Euclidean distance between key points 9 and 10 and the Euclidean distance between key points 5 and 6 is calculated as the wrist-shoulder ratio of the reserved visible light image. The average of the wrist-shoulder ratios of all reserved visible light images is calculated and denoted as a3; that is, the trunk wrist-shoulder ratio of the normal person is obtained through the above method.

[0124] When the person secretly blocks the smoking behavior, the body is curled inward, and the wrist key point is lifted, and the distance between the wrist key point and the same side shoulder key point is significantly shortened.

[0125] For each reserved visible light image, the Euclidean distance between key points 10 and 6 is calculated, and the Euclidean distance between key points 9 and 5 is calculated, and the average of the two distances is obtained as the wrist-shoulder distance of the reserved visible light image. The average of the wrist-shoulder distances of all reserved visible light images is calculated and denoted as d; both sides are calculated to avoid left-handedness, so that whether or not left-handed can be represented by the change of the distance average.

[0126] For each image of the recording camera, all reserved joint key points of the suspicious person are obtained, and according to the key point serial number of the suspicious person, the corresponding ai (i ranges from 1 to 3) value and d value are obtained. For example: if the suspicious person does not obtain key point 9, a3 does not participate in the following calculation and comparison.

[0127] After obtaining all the reserved joint key points of the suspicious person, the corresponding a1 value of the suspicious person is obtained through the key points with the same serial number, denoted as k-a1, and similarly k-a2, k-a3, and k-d values are obtained.

[0128] a1, a2, a3, and d are denoted as reference smoking parameters, and k-a1, k-a2, k-a3, and k-d are denoted as actual smoking parameters.

[0129] Step S004: Based on the reference smoking parameters and the actual smoking parameters, the key point gathering trend of the suspicious person is obtained.

[0130] In this embodiment, the torso contraction ratio, the torso elbow-shoulder ratio, the torso wrist-shoulder ratio, and the torso wrist-shoulder distance of the suspicious person are determined as actual smoking parameters; each parameter in the actual smoking parameters is compared with a corresponding parameter in the reference smoking parameters to obtain a ratio sequence of the actual smoking parameters and the reference smoking parameters; a ratio of the number of elements less than 1 to the number of all elements in the ratio sequence is denoted as n; the mean of the elements less than 1 is denoted as j; and a product of n and the -j power of the base of the natural logarithm is determined as the key point gathering trend of the suspicious person.

[0131] Specifically, when the suspicious person has corresponding smoking parameters, the ratio of the actual smoking parameters to the reference smoking parameters is calculated to obtain a plurality of ratios, and the ratios form a ratio sequence.

[0132] The more the ratios less than 1 in the ratio sequence, the greater the degree less than 1, and the greater the key point gathering trend of the suspicious person, the greater the probability of secretly smoking.

[0133] The number of ratios less than 1 in the ratio sequence is m, the number of all ratios is w, the ratio of m to w is denoted as n, the mean of the ratios less than 1 is denoted as j, and n and the -j power of the base of the natural logarithm are taken as the key point gathering trend q. The greater the value, the greater the trend, and the more likely the change of the joint key points of the suspicious person to conform to the behavior of the person secretly smoking.

[0134] Step S005: Obtain the motion periodicity of the suspicious person by using the distance change of the reserved joint key points of the suspicious person in the visible light image of the suspicious person.

[0135] In this embodiment, for the visible light images of the suspicious person in the continuous frames, the Euclidean distance between the right wrist key point and the nose key point in the reserved joint key points of the suspicious person is calculated, and a sequence formed by all the Euclidean distances is determined as a distance sequence. The distance sequence is subjected to STL decomposition to obtain a periodic term. The periodic term is subjected to Fourier transform to obtain a frequency spectrum corresponding to the periodic term. The frequency corresponding to the maximum amplitude in the frequency spectrum is determined as a main frequency, and the ratio of the maximum amplitude corresponding to the main frequency to the sum of all other amplitudes is determined as the motion periodicity of the suspicious person.

[0136] It should be noted that in the typical action of smoking underground, the person will repeatedly hold the cigarette to the mouth and suck, and then move away, which constitutes a clear periodic motion. From the perspective of human joint key points, this periodic motion directly manifests as the regular change of the distance between the wrist key point and the nose key point: when the cigarette is held to the mouth, the distance between the two points is significantly shortened; when the hand is put down or moved away, the distance increases accordingly. This kind of space distance change with stable frequency and amplitude, forms a unique pattern that is different from other limb activities.

[0137] Specifically, for each visible light image of the suspicious person, the Euclidean distance between the key points 10, 0 is calculated, and then the distances in the consecutive frames form a sequence, denoted as distance sequence.

[0138] If the distance sequence presents periodic fluctuations, the stronger the periodicity, the more the behavior of the corresponding suspicious person conforms to the smoking motion periodicity, and the greater the probability of secretly smoking.

[0139] The smoking action is not a completely standard periodic motion, and there are individual differences and random fluctuations in each cycle. For this purpose, the STL (Seasonal and Trend decomposition using Loess) decomposition method is used to decompose the key point motion sequence into trend items, periodic items and residual items. By extracting the periodic items, the noise generated by the non-standard action in different cycles is filtered out, so that the periodic pattern of the essence of smoking behavior can be more clearly represented. Finally, based on the periodicity of the cycle, the periodicity intensity analysis is carried out, and a more robust and accurate recognition of the smoking action is realized.

[0140] The distance sequence is subjected to STL decomposition to obtain the periodic item, which is a sequence of the same length as the distance sequence. The data in the periodic item represents the periodicity characteristics of the distance sequence.

[0141] The periodic item is subjected to Fourier transform to convert to the frequency domain and obtain its frequency spectrum (frequency-amplitude distribution).

[0142] The frequency corresponding to the maximum amplitude is recorded as the main frequency, and the ratio t of the main frequency amplitude to the sum of all amplitudes is calculated. The larger the ratio, the more the behavior of the suspicious person conforms to the smoking behavior, and the greater the probability that the suspicious person is secretly smoking.

[0143] Step S006: According to the key point aggregation trend and motion periodicity of the suspicious person, the smoking probability of the suspicious person is determined.

[0144] In this embodiment, the product of the key point aggregation trend and the motion periodicity of the suspicious person is determined as the smoking probability of the suspicious person.

[0145] Specifically, the product of the q value and the t value of the suspicious person is calculated as the smoking probability P of the suspicious person.

[0146] Step S007: Based on the smoking probability of the suspicious person, the suspicious person is warned.

[0147] According to the classification warning according to the smoking probability, if necessary, on-site intervention and evidence collection are carried out to ensure safety management. Specifically:

[0148] Classification warning: If 0.4 < P < 0.7, mark as "suspected event", record in the background of the system, but do not trigger real-time alarm, for the security officer to follow-up review. If P >= 0.7, immediately trigger real-time warning.

[0149] Warning information push: alarm information (including personnel ID, location, time, confidence and key evidence screenshot / video clip) is pushed to the safety production monitoring center in real time through a special network.

[0150] On-site intervention and evidence collection:

[0151] On-site sound and light alarm: near the place where the violation behavior occurs, automatically trigger the sound and light alarm device (such as a buzzer, warning light), conduct on-site real-time deterrence and intervention, and immediately stop the violation behavior.

[0152] Evidence chain preservation: the system automatically saves the high-definition video clip of a period of time (such as 30 seconds before the alarm to 10 seconds after the alarm) before and after the alarm event to the database, and generates an unalterable log record, forming a complete evidence chain.

[0153] The embodiment realizes the detection of the smoking behavior in the mine by analyzing the abnormal changes of the key points of the smoker and the periodic action mode in the smoking process. The method discards the high calculation scheme of the traditional neural network model direct recognition, and significantly improves the efficiency and real-time performance of the monitoring system.

[0154] The embodiment significantly improves the reliability of the recognition result by amplifying the behavior characteristics of the smoker instead of relying on visual detection of cigarettes. Secondly, compared with the black box characteristics of the traditional neural network, the method based on the explainable behavior feature analysis has stronger theoretical explainability. Finally, when the algorithm needs to be optimized for different special working conditions in the coal mine, this feature engineering-based method has higher adjustability and adaptability than the end-to-end neural network model.

[0155] The embodiment obtains the key point aggregation trend of the suspicious person by comparing the key point distribution of the suspicious person with the key point distribution of the normal person in the mine. The reciprocating motion of the key points of the suspicious person is obtained by analyzing the video frames of the recording camera, and the periodicity of the motion is obtained. Finally, the smoking probability is obtained by combining the key point aggregation trend and the motion periodicity.

[0156] Thus, the present application is completed.

[0157] To sum up, in the embodiment of the present application, by analyzing the thermal infrared image, the heat source and the suspicious person in the image are identified, then by analyzing the abnormal change characteristics of the human body key points in the visible light image of the suspicious person, the smoking probability of the suspicious person is determined, and finally the safety warning of the suspicious person is carried out according to the smoking probability. The present application can improve the accuracy of safety warning in coal mine underground.

[0158] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A coal mine underground safety early warning method based on semantic recognition, characterized in that, The method comprises the following steps: Collecting a set of thermal infrared images of a coal mine underground; determining a heat source region and a suspicious person region corresponding to the heat source region by using difference information of adjacent two thermal infrared images; Obtaining a reserved joint key point of the suspicious person according to position information of the suspicious person region; determining a recording camera of the suspicious person based on positions and quantities of the reserved joint key points of the suspicious person; and obtaining a visible light image of the suspicious person by using the recording camera of the suspicious person; Obtaining a reference smoking parameter according to distances between the reserved joint key points of the normal person in the visible light image of the normal person; and obtaining an actual smoking parameter according to distances between the reserved joint key points of the suspicious person in the visible light image of the suspicious person; Obtaining a key point gathering trend of the suspicious person based on the reference smoking parameter and the actual smoking parameter; The specific steps of obtaining the key point gathering trend of the suspicious person based on the reference smoking parameter and the actual smoking parameter comprise the following steps: Determining a trunk contraction ratio, a trunk elbow shoulder ratio, a trunk wrist shoulder ratio and a trunk wrist shoulder distance of the suspicious person as the actual smoking parameter; Obtaining a ratio sequence of the actual smoking parameter and the reference smoking parameter by comparing each parameter in the actual smoking parameter with a corresponding parameter in the reference smoking parameter; Recording a ratio of a number of elements less than 1 in the ratio sequence to a number of all elements as n; and recording an average of the elements less than 1 as j; Determining a product of n and a-j power of a base of a natural logarithm as the key point gathering trend of the suspicious person; Obtaining a motion periodicity of the suspicious person by using distance change conditions of the reserved joint key points of the suspicious person in the visible light image of the suspicious person; The specific steps of obtaining the motion periodicity of the suspicious person by using the distance change conditions of the reserved joint key points of the suspicious person in the visible light image of the suspicious person comprise the following steps: For consecutive frame visible light images of the suspicious person, calculating a Euclidean distance between a right wrist key point and a nose key point in the reserved joint key points of the suspicious person, and determining a sequence formed by all the Euclidean distances as a distance sequence; Performing STL decomposition on the distance sequence to obtain a periodic term; Performing Fourier transform on the periodic term to obtain a frequency spectrum corresponding to the periodic term; Determining a frequency corresponding to a maximum amplitude in the frequency spectrum as a main frequency, and determining a ratio of a maximum amplitude corresponding to the main frequency to a sum of all other amplitudes as the motion periodicity of the suspicious person; Determining a smoking probability of the suspicious person according to the key point gathering trend and the motion periodicity of the suspicious person; The specific steps of determining the smoking probability of the suspicious person according to the key point gathering trend and the motion periodicity of the suspicious person comprise the following steps: Determining a product of the key point gathering trend and the motion periodicity of the suspicious person as the smoking probability of the suspicious person; Classifying and warning the suspicious person based on the smoking probability of the suspicious person.

2. The coal mine underground safety early warning method based on semantic recognition according to claim 1, characterized in that, The specific steps of determining the heat source region and the suspicious person region corresponding to the heat source region by using difference information of adjacent two thermal infrared images comprise the following steps: Judging whether pixel values at the same positions of the adjacent two thermal infrared images are different; In the case that the pixel values at the same positions of two adjacent frames of thermal infrared images are different, a frame difference image of the two adjacent frames of thermal infrared images is obtained; The frame difference image is subjected to threshold segmentation to obtain a binary image; The binary image is subjected to morphological opening operation to obtain a denoised image; The denoised image is subjected to connected domain analysis to obtain all connected domains; The area and temperature of each connected domain are obtained, and a connected domain with an area within a preset area threshold range and a temperature greater than a preset temperature threshold is determined as a heat source region; Each connected domain outside the connected domain corresponding to the heat source region in the denoised image is determined as a candidate personnel region; For each heat source region, the minimum Euclidean distance between the current heat source region and each candidate personnel region is calculated, and a candidate personnel region with a minimum Euclidean distance less than a preset distance threshold is determined as a suspicious personnel region corresponding to the current heat source region.

3. The method for early warning of underground coal mine safety based on semantic recognition according to claim 1, characterized in that, The specific steps of obtaining the reserved joint key points of the suspicious personnel according to the position information of the suspicious personnel region include the following: According to the actual position information of the suspicious personnel region, all reserved cameras of the suspicious personnel are determined. The image of the suspicious personnel region is obtained by using each reserved camera. According to the coordinate information of the suspicious personnel in the image of the suspicious personnel region, the reserved joint key points of the suspicious personnel are obtained.

4. The coal mine underground safety early warning method based on semantic recognition according to claim 3, characterized in that, The specific steps of determining all reserved cameras of the suspicious personnel according to the actual position information of the suspicious personnel region include the following: For each suspicious personnel region corresponding to a heat source region, the actual ground coordinate points of the suspicious personnel region are obtained. According to the Euclidean distances between the actual ground coordinate points of the suspicious personnel region and the installation positions of each visible light camera, all reserved cameras of the suspicious personnel are determined.

5. The method for early warning of underground coal mine safety based on semantic recognition according to claim 3, characterized in that, The specific steps of obtaining the reserved joint key points of the suspicious personnel according to the coordinate information of the suspicious personnel in the image of the suspicious personnel region include the following: The coordinate points of the suspicious personnel in the image of the suspicious personnel region are determined as suspicious personnel coordinate points. Based on the suspicious personnel coordinate points, a lightweight model is used to detect the image of the suspicious personnel region to obtain the bounding box of the suspicious personnel. The contour of the bounding box of the suspicious personnel is extracted to obtain the contour region of the suspicious personnel. The contour region of the suspicious personnel is input into a human pose estimation model to obtain the joint key points and the confidence of the suspicious personnel. The joint key points with a confidence less than a preset confidence threshold are removed to obtain the reserved joint key points of the suspicious personnel.

6. The method for early warning of underground coal mine safety based on semantic recognition according to claim 1, characterized in that, The specific steps of determining the recording camera of the suspicious personnel based on the positions and quantities of the reserved joint key points of the suspicious personnel include the following: The reserved joint key points of the suspicious personnel are divided into first key points and second key points according to the positions of the reserved joint key points of the suspicious personnel; wherein the first key points include the key points at the head joints and the key points at the hand joints, and the second key points include the key points at the leg joints. The first key points are assigned a first weight, and the second key points are assigned a second weight; wherein the first weight is greater than the second weight. The quantity of the first key points and the quantity of the second key points are obtained. Sum the product of the number of first key points and the first weight, and the product of the number of second key points and the second weight to obtain the observation degree of the suspicious person region image; Determine the average of the observation degrees of all suspicious person region images corresponding to each reserved camera as the observation degree of each reserved camera, and determine the reserved camera corresponding to the maximum observation degree as the recording camera of the suspicious person.

7. The method for early warning of underground coal mine safety based on semantic recognition according to claim 1, characterized in that, The reference smoking parameter is obtained according to the distance between the reserved joint key points of the normal person in the visible light image of the normal person, and the specific steps include the following: Obtain the contour region of the normal person in the visible light image of the normal person; Calculate the ratio of the circumscribed rectangle area of the contour region of the normal person to the convex hull area formed by the reserved joint key points of the normal person, determine the shrinkage ratio of each visible light image of the normal person as the ratio, and determine the average of the shrinkage ratios of all visible light images of the normal person as the trunk shrinkage ratio of the normal person; Determine the ratio of the distance between the left and right elbow key points to the distance between the left and right shoulder key points in the reserved joint key points of the normal person as the elbow-shoulder ratio of each visible light image of the normal person, and determine the average of the elbow-shoulder ratios of all visible light images of the normal person as the trunk elbow-shoulder ratio of the normal person; Determine the ratio of the distance between the left and right wrist key points to the distance between the left and right shoulder key points in the reserved joint key points of the normal person as the wrist-shoulder ratio of each visible light image of the normal person, and determine the average of the wrist-shoulder ratios of all visible light images of the normal person as the trunk wrist-shoulder ratio of the normal person; Determine the average of the distance between the right wrist and the right shoulder key points and the distance between the left wrist and the left shoulder key points in the reserved joint key points of the normal person as the wrist-shoulder distance of each visible light image of the normal person, and determine the average of the wrist-shoulder distances of all visible light images of the normal person as the trunk wrist-shoulder distance of the normal person; Determine the trunk shrinkage ratio, the trunk elbow-shoulder ratio, the trunk wrist-shoulder ratio, and the trunk wrist-shoulder distance of the normal person as the reference smoking parameter.

Images