Method and device for identifying and alarming safety helmet under mine and computing equipment

Abstract

The invention provides a method and a device for identifying and alarming a safety helmet under a mine and computing equipment, the method comprises the following steps: acquiring a frame image of a real-time video stream from a camera device, the real-time video stream being used for fixedly monitoring a working place under the mine; performing human body detection identification on the frame image to obtain a human body identification area; performing head area positioning on the human body recognition area to obtain a head recognition area; carrying out safety helmet wearing identification detection on the head identification area to obtain a safety helmet wearing identification result; and rechecking the safety helmet wearing identification result based on a semantic segmentation model, and determining whether to output a safety helmet non-wearing alarm signal or not according to a rechecking result. According to the technical scheme of the invention, accurate safety helmet wearing identification can be realized for mines with complex environments and under-mine environments.

Description

Technical Field

[0001] The present invention relates to the technical field of safety production identification in mines, and in particular to a method and device for identifying and alarming a safety helmet in mines, and computing equipment. Background Art

[0002] In mining operations, hard hats are essential personal protective equipment for workers, which can effectively protect the head from falling objects, collisions and other dangers. However, due to the complex environment and harsh working conditions in mines, workers sometimes forget to wear hard hats or deliberately do not wear them, which brings potential risks to mine safety. According to statistics, head injuries caused by not wearing hard hats account for a large proportion of mine accidents each year. These accidents not only cause casualties, but may also lead to production interruptions and economic losses.

[0003] In recent years, with the development of artificial intelligence technology, the technology for identifying people who are not wearing helmets has been constantly updated. Most of the existing deep learning technologies are based on target detection or target detection and target recognition technology. In non-mining environments, the color characteristics of construction site helmets are obvious, and the strong light during the day makes the image clear and easy to identify. The existing recognition technologies can achieve good recognition accuracy. Due to the complex scenes in the mines, the fluorescent light environment causes the light in the mines to be dark and the imaging to be poor. The color characteristics of the helmets in the mines are dark blue and gray, which makes the recognition difficult. The existing recognition technologies often have low recognition accuracy and high false detection rates.

[0004] To this end, a technical solution is needed to achieve accurate helmet wearing identification in complex mine and underground environments. Summary of the invention

[0005] The present invention aims to provide a method and device for identifying and alarming safety helmets in mines, and computing equipment, so as to realize accurate identification of safety helmet wearing in mines and underground environments with complex environments.

[0006] According to one aspect of the present invention, a method for identifying and alarming a safety helmet in a mine is provided, comprising:

[0007] Acquire a frame image of a real-time video stream from a camera device, wherein the real-time video stream is used for fixed monitoring of a workplace in a mine;

[0008] Performing human body detection and recognition on the frame image to obtain a human body recognition area;

[0009] Positioning the head region of the human body recognition area to obtain the head recognition area;

[0010] Performing a helmet wearing recognition detection on the head recognition area to obtain a helmet wearing recognition result;

[0011] The helmet wearing recognition result is reviewed based on the semantic segmentation model, and it is determined whether to output a helmet not wearing alarm signal according to the review result.

[0012] According to some embodiments, performing human body detection and recognition on the frame image to obtain a human body recognition area includes:

[0013] Performing human body detection and recognition on the frame image by using a human body detector to obtain a human body target frame and a detection score for each person in the frame image;

[0014] If the detection score is greater than a set first threshold, the corresponding human target frame is determined to be valid, and the human target frame is used as the human recognition area for further recognition.

[0015] According to some embodiments, before performing human body detection and recognition on the frame image to obtain the human body recognition area, the method further includes:

[0016] Adjusting the input size of the frame image;

[0017] The human detector is trained using a mine pedestrian dataset.

[0018] According to some embodiments, performing head region positioning on the human body recognition area to obtain the head recognition area includes:

[0019] The head region of the human body recognition area is located by a head and shoulder key point detector network to obtain the head and shoulder key points and the visible attributes, key point coordinates and corresponding confidence scores of the head and shoulder key points, wherein the head and shoulder key points include: head vertex, neck point, right shoulder point and left shoulder point;

[0020] If the confidence score of the key point is greater than a second threshold, the head recognition area is obtained according to the coordinates of the head and shoulder key points.

[0021] According to some embodiments, performing a helmet wearing recognition detection on the head recognition area to obtain a helmet wearing recognition result includes:

[0022] If the head vertex visibility attribute of the head recognition area is less than a third threshold, performing a helmet wearing recognition detection on the head recognition area;

[0023] Performing a helmet wearing recognition detection on the head recognition area to obtain a first probability score of not wearing a helmet;

[0024] If the first probability score is greater than a fourth threshold, it is preliminarily determined that the person is not wearing a helmet.

[0025] According to some embodiments, the human detector adopts the yolov7 target detection framework, and the convolution module of the target detection framework adopts the leakyRelu activation layer; the head and shoulder key point detector network is constructed by combining the HRNet network with the residual structure of ResNet18; the helmet wearing recognition detection adopts the residual structure of ResNet50 network structure.

[0026] According to some embodiments, reviewing the helmet wearing recognition result based on the semantic segmentation model to determine whether the helmet wearing recognition result is accurate includes:

[0027] Inputting the head recognition area determined as not wearing a helmet into the semantic segmentation model, wherein the segmentation categories of the semantic segmentation model include: skin, hair, background, and helmet;

[0028] When the skin contour and hair contour are segmented, and the helmet contour is smaller than the hair contour, it is judged that the helmet is not worn and a warning is issued.

[0029] According to another aspect of the present invention, a device for identifying and alarming safety helmets in mines is provided, the device comprising: an acquisition module, a human body detection identifier, a head identification locator, and a safety helmet-not-wearing detector, wherein:

[0030] The acquisition module is used to acquire frame images of a real-time video stream from a camera device, and the real-time video stream is used for fixed monitoring of the underground mine workplace;

[0031] The human body detection and recognition device is used to perform human body detection and recognition on the frame image to obtain a human body recognition area;

[0032] The head recognition locator locates the head region of the human body recognition area to obtain the head recognition area;

[0033] The non-helmet-wearing detector performs helmet-wearing recognition detection on the head recognition area to obtain a helmet-wearing recognition result;

[0034] The region segmentation module reviews the helmet wearing recognition result based on the semantic segmentation model, and determines whether to output a helmet not wearing alarm signal according to the review result.

[0035] According to another aspect of the present invention, there is provided a computer program product, comprising: a computer program, wherein when the computer program is executed by a processor, the method described in any one of the above is implemented. According to another aspect of the present invention, there is provided a computing device, comprising:

[0036] Processor; and

[0037] A memory storing a computer program, which, when executed by the processor, enables the processor to execute any of the methods described above.

[0038] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, and when the instructions are executed by a processor, the processor is caused to execute any of the above methods.

[0039] According to the embodiments of the present invention, the human body recognition area is obtained by implementing target detection by applying a human body detector. Compared with directly detecting the head area in the prior art, the human body detection has better detection effect and is less likely to miss detection; the human head and shoulder key points of the human body recognition area are located by a head and shoulder key point detector network to indirectly obtain the head recognition area and reduce misidentification; the wearing of a safety helmet is preliminarily judged and reviewed by wearing a safety helmet recognition detection and a semantic segmentation model, and the result of not wearing a safety helmet that has been identified by wearing a safety helmet recognition detection is further judged by semantic segmentation to prevent false alarms caused by misidentification. The false alarm rate is further reduced by the double judgment method.

[0040] According to some embodiments, the head and shoulder key points of the human body recognition area are located through a head and shoulder key point detector network. The key point positioning includes a total of 4 points, which reduces redundant calculations and improves recognition efficiency compared to the traditional 21-point method. Afterwards, the head recognition area is calculated by two points, the top of the head and the neck. Moreover, by performing a threshold judgment on the confidence of the key points, false detections caused by human body detection can be filtered out, and by performing a threshold judgment on the visibility attributes of the key points, a part of the situation where a helmet is worn can be filtered out to reduce false recognition.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0043] Figure 1 A flow chart of a method for identifying and alarming a safety helmet in a mine according to an exemplary embodiment is shown.

[0044] Figure 2 A schematic diagram of a head and shoulder key point detector is shown in a method for identifying and alarming a safety helmet in a mine according to an example embodiment.

[0045] Figure 3 A schematic diagram of head and shoulder key point detection of a method for identifying and alarming a safety helmet in a mine according to an example embodiment is shown.

[0046] Figure 4 A schematic diagram of semantic segmentation of a method for identifying and alarming a safety helmet in a mine according to an example embodiment is shown.

[0047] Figure 5 A schematic diagram of semantic segmentation of a method for identifying and alarming a safety helmet in a mine according to another exemplary embodiment is shown.

[0048] Figure 6 A schematic flow chart of a method for identifying an alarm for a safety helmet in a mine according to an exemplary embodiment is shown.

[0049] Figure 7 A schematic diagram of a head recognition area of ​​a method for identifying and alarming a safety helmet in a mine according to an exemplary embodiment is shown.

[0050] Figure 8 A schematic diagram of a device for identifying and alarming a safety helmet in a mine according to an exemplary embodiment is shown.

[0051] Fig. 9 A block diagram of a computing device is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0053] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0055] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0056] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the present inventive concept. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0057] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention, and therefore cannot be used to limit the protection scope of the present invention.

[0059] In recent years, with the development of artificial intelligence technology, the technology for identifying people not wearing helmets has been constantly updated. Most of the existing deep learning technologies are based on target detection or target detection and target recognition technology.

[0060] In mining operations, hard hats are essential personal protective equipment for workers, which can effectively protect the head from falling objects, collisions and other dangers. However, due to the complex environment and harsh working conditions in mines, workers sometimes forget to wear hard hats or deliberately do not wear them, which brings potential risks to mine safety. According to statistics, head injuries caused by not wearing hard hats account for a large proportion of mine accidents each year. These accidents not only cause casualties, but may also lead to production interruptions and economic losses. In non-mining environments, the color characteristics of construction site hard hats are obvious, and the strong light during the day makes the image clear and easy to identify. Existing recognition technologies can have a good recognition accuracy rate. Due to the complex scenes in the mine, the fluorescent light environment causes the light in the mine to be dark and the imaging is poor. The color characteristics of the hard hats in the mine are dark blue and gray, which makes it difficult to identify. Existing recognition technologies often have low recognition accuracy and high false detection rate.

[0061] To this end, the present invention proposes a method and computing device for identifying and alarming safety helmets in mines with complex environments and underground environments, so as to realize accurate identification of safety helmet wearing. According to the embodiment, the design scheme of the present invention obtains the human body identification area by applying a human body detector to realize target detection. Compared with directly detecting the head area in the prior art, the human body detection has better detection effect and is less likely to miss detection; the human head and shoulder key points of the human body identification area are located by a head and shoulder key point detector network to indirectly obtain the head identification area and reduce misidentification; the wearing of safety helmets is preliminarily judged and reviewed by wearing safety helmet identification detection and semantic segmentation model, and the results of not wearing safety helmets that have been identified by wearing safety helmet identification detection are further judged by semantic segmentation to prevent false alarms caused by misidentification. The false alarm rate is further reduced by the double judgment method.

[0062] According to some embodiments, the head and shoulder key points of the human body recognition area are located through the head and shoulder key point detector network. The key point positioning includes 4 points in total. Compared with the traditional 21-point method, redundant calculations are reduced and recognition efficiency is improved. Afterwards, the head recognition area is obtained by calculating the top of the head and the neck. Moreover, by performing a threshold judgment on the confidence of the key points, false detections caused by human body detection can be filtered out, and by performing a threshold judgment on the visibility attributes of the key points, a part of the situation where a helmet is worn can be filtered out to reduce false recognition.

[0063] Before describing the embodiments of the present invention, some terms or concepts involved in the embodiments of the present invention are explained.

[0064] Yolov7 is the latest version of the YOLO (You Only Look Once) series of target detection algorithms, which is a real-time object recognition and positioning technology based on deep neural networks.

[0065] Silu (Sigmoid Linear Unit) and LeakyReLU (Leaky Rectified Linear Unit) are both activation functions used in neural networks, and each has unique characteristics and application scenarios. Silu, also known as Swish, is a self-gated activation function proposed by researchers at Google. However, ReLU has a disadvantage that it outputs 0 for all negative inputs, which may cause some neurons to "die", that is, these neurons no longer respond to any input data, thus hindering the training process. LeakyReLU solves this problem by introducing a small slope so that there is a non-zero output for negative inputs.

[0066] HRNet (High-Resolution Network) is a neural network architecture used for computer vision tasks, especially image segmentation and pose estimation.

[0067] Exemplary embodiments of the present invention are described below with reference to the accompanying drawings.

[0068] Figure 1 A flow chart of a method for identifying and alarming a safety helmet in a mine according to an exemplary embodiment is shown.

[0069] See also Figure 1 , the figure shows a method for identifying and alarming a safety helmet in a mine, and the specific steps of the method are as follows:

[0070] In S101, a frame image of a real-time video stream from a camera device is acquired, where the real-time video stream is used for fixed monitoring of a working place in a mine.

[0071] According to some embodiments, first, a real-time video stream is obtained from a camera installed in a mine and decomposed into a series of frame images. The camera is usually arranged in a key position to ensure that all important working areas can be covered, so as to achieve effective monitoring of the underground mine environment. In order to ensure the quality and stability of the video, the camera should have good low-light adaptability and anti-interference performance.

[0072] In S103, human body detection and recognition is performed on the frame image to obtain a human body recognition area.

[0073] According to some embodiments, computer vision technology is used to detect and identify the human body and generate a human body identification area. For example, a target detection model based on deep learning is used, such as the YOLO series, Faster R-CNN, etc. By applying the target detection model, the position of the human body in the picture can be quickly and accurately located. Compared with directly detecting the human head area, the advantage of human body detection is that its target is larger, which reduces the problem of missed detection of small targets due to excessive distance.

[0074] In S105, the head region of the human body recognition area is located to obtain a head recognition area.

[0075] According to some embodiments, after determining the position of the human body recognition area, the next step is to accurately locate the head area. By detecting the key points of the human body (such as the top of the head, neck, and shoulders) to indirectly infer the position of the head and shoulders, instead of directly applying the target detection method in the prior art, this method can not only find the head more accurately, but also use the confidence and visibility attributes of the key points to filter out false detections, further improving the accuracy of the system. For the problems of occlusion or uneven lighting that may exist in the mine environment, the key point detection method can be used to further reduce the possibility of false detection and improve the overall robustness of the method.

[0076] In S107, the head recognition area is subjected to a helmet wearing recognition detection to obtain a helmet wearing recognition result.

[0077] According to some embodiments, after obtaining the head recognition area, the next step is to identify whether a helmet is worn in the area, which can be accomplished using a specially trained helmet classifier that can distinguish between wearing and not wearing states, and perform state probability analysis on the head recognition area to obtain a probability score for the corresponding state. The obtained probability score is judged by setting a threshold to determine whether the helmet is not worn.

[0078] In S109, the helmet wearing recognition result is reviewed based on the semantic segmentation model, and it is determined whether to output a helmet not wearing alarm signal according to the review result.

[0079] According to some embodiments, semantic segmentation is a task that assigns each pixel in an image to a specific category, which can provide more detailed information than a simple bounding box. By performing a secondary inspection and judgment on the image without a helmet, the false detection rate can be effectively reduced, ensuring that the final output of the helmet wearing status judgment is more reliable. This method not only takes into account the challenges in the special environment of the mine, but also uses advanced AI technology to improve detection accuracy and ensure the safety of the miners. In the entire process design, each link is closely linked to form an efficient and stable automatic monitoring system. Such a system is crucial to maintaining the safety of mining operations, and it also reflects the importance of science and technology serving social security.

[0080] According to some embodiments, human body detection and recognition is performed on the frame image to obtain a human body recognition area, and human body detection and recognition is performed on the frame image through a human body detector to obtain a human body target frame and a detection score for each person in the frame image; if the detection score is greater than a set first threshold, the corresponding human body target frame is judged to be valid, and the human body target frame is used as the human body recognition area for further recognition. First, a pre-trained human body detection model is used to comprehensively scan the entire image to locate all possible human body positions. The human body detection model will output a series of human body target frames, each of which not only contains the position information of the human body, but also comes with a detection score indicating the confidence level.

[0081] According to some embodiments, these detected target frames are then evaluated. Specifically, it is checked whether the detection score corresponding to each target frame exceeds a pre-set first threshold. The first threshold is used to distinguish between valid and invalid detection results. Only when the detection score is higher than the threshold, the corresponding target frame will be considered valid and then selected as the human recognition area in the subsequent steps. This method not only improves the accuracy of detection, but also enhances the robustness of the system, so that it can maintain good performance even in harsh working environments. In addition, such a design also helps to reduce unnecessary consumption of computing resources, thereby supporting longer continuous monitoring.

[0082] According to some embodiments, selecting an appropriate threshold is critical to ensuring the performance of the system. If the threshold is set too low, a large number of false positives may result; conversely, if the threshold is too high, some real but less obvious targets may be missed. Therefore, in practical applications, it is usually necessary to adjust and optimize according to specific environmental conditions and requirements, so as to effectively screen out high-quality human recognition areas and prepare for subsequent helmet wearing recognition.

[0083] According to some embodiments, the human detector adopts the yolov7 target detection framework, and the convolution module of the target detection framework adopts the leakyRelu activation layer. The human detector adopts the yolov7 target detection framework in the conventional yolo series, and is trained using a data set of pedestrians under a mine. During the training process, this patent modified the yolov7 network training process, changing the activation layer in the convolution module (convolution layer + batch normalization layer + activation layer) from the original silu activation layer to the leakyRelu activation layer to increase the speed of training the model.

[0084] According to some embodiments, before human body detection and recognition are performed on the frame image and the human body recognition area is obtained, the input size of the frame image can be adjusted, and the human body detector can be trained using a dataset of pedestrians in a mine. In order to adapt to different application scenarios, especially in situations where non-standard resolution images may exist, such as mines, another dimension is automatically adjusted to maintain the aspect ratio to avoid image deformation. If the aspect ratio of the original image does not match the target size, the actual width or height of the output image may be less than the specified value. Adjust the network input size, and adjust the original 416*416 to 512*512 size, in order to balance speed while improving the accuracy of human target detection.

[0085] According to some embodiments, the underground pedestrian dataset may be re-clustered to obtain anchor frame coordinates adapted to the underground mine scene. For example, the underground mine dataset is used to train the human detector, and the underground pedestrian dataset is re-clustered according to the human coordinate frames known in the training set using a k-means clustering algorithm to obtain anchor frame coordinates adapted to the underground mine scene. Re-clustering the training data to obtain anchor frame coordinates adapted to the scene makes the target positioning of the human detection model more accurate and more adapted to the applied scene.

[0086] According to some embodiments, in order to make Yolov7 better adapt to the pedestrian detection task in the underground mine environment, it is necessary to use a dataset specially collected for this scene for training. These datasets should contain a large number of real mine internal images with pedestrian positions marked so that the model can learn unique visual features. Considering the uniqueness of the mine environment, such as poor lighting conditions and large changes in viewing angles, directly applying a general dataset may not achieve ideal detection results. Therefore, it is very important to re-cluster the dataset for the underground mine scene to obtain adaptive anchor frame coordinates.

[0087] According to some embodiments, first, a high-quality dataset of pedestrians under the mine needs to be prepared, which can be done by collecting on-site pictures and using tools such as Labelme for precise labeling. When labeling, make sure that each pedestrian is accurately marked to form a bounding box. In addition, a list of training set and validation set files needs to be generated, and the corresponding data path needs to be configured. Afterwards, an anchor frame clustering operation is performed based on the prepared dataset. Yolov7 uses a set of predefined prior frames by default, but in specific applications, these prior frames may not be completely suitable. Through the K-means algorithm or other suitable clustering methods, a set of anchor frame coordinates that are more in line with the characteristics of pedestrians under the mine can be recalculated based on the size distribution of objects in the actual dataset. This process helps to improve the detection accuracy of the model in specific scenarios. Finally, after completing the dataset preparation and anchor frame clustering, you can start training the Yolov7 model.

[0088] According to some embodiments, the convolution module of yolov7 uses Leaky ReLU as the activation function instead of the traditional ReLU. Leaky ReLU allows a certain slope even for negative inputs to prevent the emergence of "dead neurons", thereby promoting better gradient flow and model convergence. During the training process, other optimization strategies, such as adaptive learning rate adjustment, data enhancement techniques, etc., can also be considered to further improve model performance. By standardizing the input size of frame images and conducting targeted training using data sets designed specifically for underground mine environments, including re-clustering to obtain more adaptable anchor frame coordinates, the yolov7-based human detector can show higher robustness and accuracy in complex and changeable mine environments. Such a customized solution not only improves detection efficiency, but also provides strong technical support for ensuring the safety of miners.

[0089] According to some embodiments, the head region of the human body recognition area is positioned, and when the head recognition area is obtained, the head region of the human body recognition area is positioned by a head and shoulder key point detector network to obtain head and shoulder key points and visible attributes, key point coordinates and corresponding confidence scores of the head and shoulder key points, wherein the head and shoulder key points include: head vertex, neck point, right shoulder point, left shoulder point; if the confidence score of the key point is greater than a second threshold, the head recognition area is obtained according to the head and shoulder key point coordinates. The head and shoulder key point detector network is applied to the human body recognition area image that has been confirmed by the human body detector to extract specific key points related to the head and shoulders, and the visible attributes, coordinates and corresponding confidence scores of each head and shoulder key point are obtained from the detection results, and the key points include but are not limited to the head vertex, neck point, right shoulder point and left shoulder point. Among them, the visible attribute indicates whether the key point is clearly visible in the current view; the coordinates indicate the position of the key point in the image; and the confidence score reflects the accuracy of the model's prediction of the key point position. The confidence score of each key point is checked. If the confidence score of a key point is greater than the set second threshold, the key point is considered to be reliable and is used for subsequent processing.

[0090] Figure 2 A schematic diagram of a head and shoulder key point detector is shown in a method for identifying and alarming a safety helmet in a mine according to an example embodiment.

[0091] According to some embodiments, the head and shoulder key point detector network is constructed by using a HRNet network combined with a residual structure ResNet18. Figure 2, the figure uses a two-branch head and shoulder key point detector network, one branch outputs 4 key point coordinates and corresponding confidence scores, and the other branch outputs the visibility attributes corresponding to the 4 key points. Among them, the visibility attributes are visible and invisible. Point visibility means that the point is not blocked, represented by the number 0, and invisible means that the current point is blocked, represented by the number 1. One of the two branches uses the HRNet network as the backbone of the key point positioning network in the head and shoulder key point detection. The other visibility attribute branch has two HRNet convolutional layers, which are connected to the residual structure ResNet18 for the visibility attribute prediction of the head and shoulder key points. The structures of the two branches are as follows: Figure 2 As shown in , block1 to 4 convolutional layers form a residual structure, and stage1 to 4 also form convolutional layers of a residual structure. By locating the head area of ​​the human body recognition area, the head position of each worker can be accurately determined, providing a basis for further analysis of the helmet wearing status and providing strong support for subsequent safety monitoring and management.

[0092] Figure 3 A schematic diagram of head and shoulder key point detection of a method for identifying and alarming a safety helmet in a mine according to an example embodiment is shown.

[0093] According to some embodiments, see Figure 3 The head and shoulder key points mainly define four points, namely the head vertex, neck point, right shoulder point, and left shoulder point. Among them, the visibility attribute of the point uses 0 or 1 to indicate whether it is visible. The training data set label contains the coordinates of the four key points and the corresponding visibility attributes, for example, (x1, y1, 0), (x2, y2, 0), (x3, y3, 0), (x4, y4, 1). For the head vertex with a helmet, when marking the visibility attribute, it is regarded as being blocked by the helmet and marked as invisible, so that the model can learn that the helmet is invisible in the subsequent learning process, and judge whether the helmet is worn by the head vertex visibility attribute.

[0094] According to some embodiments, training data marked with coordinate points and visibility attributes can be input into the above-designed two-branch network for training and learning. The two branches are trained separately. The head and shoulders key point branch is trained separately first. The training data is uniformly scaled to a size of 192*256 and sent to the network according to batch_size*3*192*256 input. The output is batch_size*4*48*64 dimensions until the training loss converges and the head and shoulders key point branch training is completed. Then, the model parameters of the head and shoulders key branch are frozen, and only the head and shoulders visibility attribute branch is trained. The same preprocessing is performed and the output is 1*4 dimensions until the training loss converges. At this time, the head and shoulders visibility attribute branch is trained, and finally a head and shoulders key point detection model with visibility attributes is obtained.

[0095] According to some embodiments, when the head recognition area is subjected to helmet wearing recognition detection and the helmet wearing recognition result is obtained, if the visible attribute of the head vertex of the head recognition area is less than the third threshold, the head recognition area is subjected to helmet wearing recognition detection; the head recognition area is subjected to helmet wearing recognition detection to obtain a first probability score of not wearing a helmet; if the first probability score is greater than the fourth threshold, it is preliminarily determined that the helmet is not worn. Specifically, the visible attribute of the head vertex in the head recognition area is checked. If the visible attribute of the head vertex is less than the set third threshold (i.e., the head vertex is not obvious or blocked), it is considered that the area is suitable for helmet wearing recognition detection. The helmet wearing recognition model is applied to the head recognition area that meets the conditions. A pre-trained deep learning model is used, for example, based on the ResNet50 structure, to classify the head in the image, and output a probability score indicating that the helmet is not worn, i.e., the first probability score, which reflects the possibility that the head is not wearing a helmet. The obtained first probability score is compared with the fourth threshold, and if the first probability score is greater than the fourth threshold, it is preliminarily determined that the helmet is not worn.

[0096] According to some embodiments, the wearing helmet recognition detection adopts the ResNet50 network structure with a residual structure. The ResNet50 network structure with a residual structure is used as the basic model. ResNet50 has sufficient depth and width, can effectively extract features from images, and has good generalization ability. The ResNet50 network structure with a residual structure is used, and the training data that has been labeled with wearing a helmet (label category is 0) and not wearing a helmet (label category is 1) are sent to the ResNet50 network for training with a network input size of 128x128 scale until the training loss converges and the identifier training is completed. The model outputs a probability score indicating that the helmet is not worn. By performing helmet wearing recognition detection on the head recognition area, the helmet wearing status of the underground workers can be effectively judged, which not only improves the accuracy of recognition, but also provides strong support for subsequent safety monitoring and management. The helmet wearing recognition detection method combines the advantages of head vertex visible attribute inspection and deep learning model classification, and can achieve efficient and accurate helmet wearing status monitoring in complex and changeable underground mine environments.

[0097] Figure 4 A segmented schematic diagram of a method for identifying an alarm for a safety helmet in a mine according to an exemplary embodiment is shown.

[0098] Figure 5 A segmented schematic diagram of a method for identifying an alarm for a safety helmet in a mine according to another exemplary embodiment is shown.

[0099] According to some embodiments, the helmet wearing recognition result is reviewed based on the semantic segmentation model to determine whether the helmet wearing recognition result is accurate, and the head recognition area determined to be not wearing a helmet is input into the semantic segmentation model. Figure 4 as well as Figure 5 The segmentation categories of the semantic segmentation model include: skin, hair, background and helmet; when the skin contour and hair contour are segmented, and the helmet contour is smaller than the hair contour, it is judged that the helmet is not worn and a warning is issued.

[0100] According to some embodiments, in order to further determine whether the result of not wearing a helmet is accurate, a segmenter based on semantic segmentation is first designed to review the result of not wearing a helmet to reduce the probability of false alarm. Specifically, the feature extraction network of the segmenter can use HNRet as the backbone, the input is 128*128, the segmentation head is connected to two convolutional layers, and a feature map of 1*4*128*128 dimensions is output, corresponding to 4 categories: binary images of helmets, hair, skin, and background. The training data of the segmenter is first annotated with an open source annotation tool to annotate the contours of 4 categories (closed areas composed of multiple points). Finally, each head area annotated category contour is converted into a corresponding closed area with different pixel values. Binary image labels (.png format). Among them, the closed area of ​​the helmet contour is filled with pixel value 1, the closed area of ​​the hair contour is filled with pixel value 2, the closed area of ​​the skin contour is filled with pixel value 3, and the remaining area is the background, filled with pixel value 0. In addition, in order to make the segmenter have better processing capabilities for images of the human head area with poor imaging quality, when making training data, the images of the human head area with poor imaging and the human head area where the naked eye cannot distinguish the helmet, hair, and skin contour are uniformly processed as the background and filled with pixels 0 as training data. This will make the segmenter more capable of processing images of the human head area with poor imaging, so as to indirectly filter out false detections. After the training data is made, the training data is input into the defined segmentation network in batch_size*3*128*128 size for training and learning until the training loss converges and the segmenter training is completed.

[0101] Figure 6 A schematic flow chart of a method for identifying an alarm for a safety helmet in a mine according to an exemplary embodiment is shown.

[0102] The figure shows a specific process of a method for identifying and alarming a safety helmet in a mine according to an exemplary embodiment, see Figure 6 , obtain a real-time video stream frame image from the monitoring camera device, send the frame image to the human detector for human target positioning, and obtain the coordinate frame and detection score of each person. Afterwards, filter the non-human target frame based on the detection score value of each human target frame. When the detection score of the human target frame is greater than the set first threshold (such as 0.5), the corresponding human target frame is retained and input into the head and shoulder key point positioning model.

[0103] The key point confidence scores are used to further determine whether the human target frame image is a person. The key point positioning branch in the head and shoulder key point positioning model outputs the key point coordinates of the four points and the confidence values ​​of the four points. The visibility attribute branch outputs the visibility scores of the four key points. The smaller the score, the greater the probability of being unobstructed, and the larger the score, the greater the probability of being obstructed. It is determined whether the confidences corresponding to the four points are all greater than the set second threshold (for example, 0.5). If they are all greater than the second threshold, it means that it is a positive detection, that is, the human target frame is determined to be a person. Otherwise, it is regarded as a false detection, and the current human target frame is filtered out, and the rest is retained as the human recognition area.

[0104] Figure 7 A schematic diagram of a head recognition area of ​​a method for identifying and alarming a safety helmet in a mine according to an exemplary embodiment is shown.

[0105] See also Figure 7 , locate the head area of ​​the human body recognition area. When the confidence of each point is greater than the threshold, the head recognition area to be recognized can be indirectly obtained through the coordinates of the head vertex and the neck. Specifically, the method for obtaining the head recognition area needs to first calculate the center point p[(x1+x2) / 2, (y1+y2) / 2] of the head vertex (x1, y1) and the neck (x2, y2), and then calculate the distance d between the two points using the following formula:

[0106]

[0107] Then, a line is drawn through point p at a distance of d / 2 to obtain pp1 and pp2, and the four minimum circumscribed rectangles of p1, p2, the neck point, and the head vertex are obtained to obtain the black rectangular frame as shown in the figure. Finally, the width and height of the black rectangular frame are enlarged by 1.5 times, and finally the final head recognition area is obtained (as shown in the figure). Figure 7 (shown in red box).

[0108] Finally, after obtaining the head recognition area, the visibility attribute score of the head vertex is used to determine whether a helmet is worn. When the visibility score is greater than the set third threshold (for example, 0.9), it means that the point is not visible, blocked by the helmet or blocked by other objects. At this time, the current head area portrait is no longer analyzed by the next model; the head area portrait with a visibility score less than the set third threshold is sent to the next model for identification without a helmet. The head recognition area retained by the previous model is sent to the non-helmet identifier, and the output is passed through softmax to obtain the probability scores of two categories. When the first probability score (non-helmet category score) is greater than the fourth threshold, it means that the helmet is not worn, otherwise the helmet is worn. When it is recognized that the helmet is not worn, the last segmenter is entered to review the results.

[0109] The segmenter receives the image of the head area without a helmet processed by the helmetless identifier and performs regional segmentation to obtain a closed contour area with a category. When the skin contour and hair contour are segmented, and the helmet contour (including the case where the helmet contour is not segmented) is smaller than the hair contour, the final result is determined to be no helmet, and the no helmet alarm signal is output at this time; if the helmet contour is segmented, it can be determined that the helmet is worn, and the no helmet alarm signal is not output; if nothing is segmented, it means that the current head area image is poorly imaged and cannot be distinguished, and the no helmet alarm signal is not output. After the segmenter analysis is completed and the corresponding result signal is output, the recognition process ends and the analysis of the next frame begins.

[0110] Figure 8 A schematic diagram of a device for identifying and alarming a safety helmet in a mine according to an exemplary embodiment is shown.

[0111] See also Figure 8 , the figure shows a device for identifying and alarming a safety helmet in a mine according to an example embodiment, the device comprising: an acquisition module 01, a human body detection identifier 02, a head recognition and locator 03, a safety helmet not wearing detector 04 and a region segmentation module 05, wherein the acquisition module 01 is used to acquire a frame image of a real-time video stream from a camera device, and the real-time video stream is used for fixed monitoring of a workplace in a mine; the human body detection identifier 02 is used to perform human body detection and identification on the frame image to obtain a human body identification area; the head recognition and locator 03 performs head area positioning on the human body identification area to obtain a head identification area; the safety helmet not wearing detector 04 performs safety helmet wearing identification detection on the head identification area to obtain a safety helmet wearing identification result; the region segmentation module 05 reviews the safety helmet wearing identification result based on a semantic segmentation model, and determines whether to output a safety helmet not wearing alarm signal according to the review result.

[0112] According to some embodiments, the design of the present invention realizes target detection by applying a human detector 02 to obtain the human recognition area, and detects all people in the frame image. Compared with the prior art of directly detecting the head area, since the human target has larger pixels than the head target, and the head target is smaller when it is beyond a certain distance, the human detection effect is better and less likely to be missed.

[0113] According to some embodiments, the head recognition locator 03 is used to locate the key points of the human head and shoulders in the human recognition area to indirectly obtain the head recognition area. The key point positioning includes: the top of the head, the neck point, the right shoulder point, and the left shoulder point, a total of 4 points, as well as the visibility attributes and confidence of the key points. Compared with the traditional 21-point positioning method, redundant calculations are reduced and recognition efficiency is improved. Afterwards, the head recognition area is obtained by calculating the two points of the top of the head and the neck. In addition, by performing a threshold judgment on the confidence of the key points, false detections caused by human body detection can be filtered out, and by performing a threshold judgment on the visibility attributes of the key points, a part of the situation where a helmet is worn can be filtered out to reduce false recognition.

[0114] According to some embodiments, the region segmentation module 04 and the region segmentation module 05 are used to make a preliminary judgment and review on whether a helmet is worn, and semantic segmentation is used to make a further judgment on the result of not wearing a helmet that has been identified by the helmet wearing recognition detection, so as to prevent false alarms caused by wrong recognition. The false alarm rate is further reduced by the double judgment method.

[0115] According to some embodiments, the design scheme of the present invention can also be applied to a computer program product. When the computer program is executed by a processor, it implements the method described in any of the above items, so that the monitoring system can realize accurate helmet wearing identification in mines with complex environments and underground mine environments.

[0116] According to some embodiments, the design scheme of the present invention obtains the human body recognition area by applying a human body detector to achieve target detection. Compared with the direct detection of the head area in the prior art, the human body detection has better detection effect and is less likely to miss detection; the human head and shoulder key points of the human body recognition area are located by a head and shoulder key point detector network to indirectly obtain the head recognition area. The key point positioning includes a total of 4 points. Compared with the traditional 21-point method, redundant calculations are reduced and recognition efficiency is improved. After that, the head recognition area is obtained by calculating the top of the head and the neck. Moreover, by thresholding the confidence of the key points, false detections generated by human body detection can be filtered, and by thresholding the visibility attributes of the key points, a part of the cases where a helmet has been worn can be filtered to reduce false recognition; by wearing a helmet recognition detection and a semantic segmentation model, a preliminary judgment and review of wearing a helmet is performed, and by semantic segmentation, the results of not wearing a helmet that have been identified by wearing a helmet recognition detection are further judged to prevent false alarms caused by false recognition. The false alarm rate is further reduced by the double judgment method.

[0117] Fig. 9 A block diagram of a computing device according to an example embodiment of the invention is shown.

[0118] like Fig. 9As shown, computing device 30 includes processor 12 and memory 14. Computing device 30 may also include bus 22, network interface 16, and I / O interface 18. Processor 12, memory 14, network interface 16, and I / O interface 18 may communicate with each other via bus 22.

[0119] The processor 12 may include one or more general-purpose CPUs (Central Processing Units, processors), microprocessors, or application-specific integrated circuits, etc., for executing relevant program instructions. According to some embodiments, the computing device 30 may also include a high-performance graphics card (GPU) 20 for accelerating the processor 12.

[0120] The memory 14 may include a machine system readable medium in the form of a volatile memory, such as a random access memory (RAM), a read-only memory (ROM) and / or a cache memory. The memory 14 is used to store one or more programs including instructions and data. The processor 12 can read the instructions stored in the memory 14 to execute the above-mentioned method according to the embodiment of the present invention.

[0121] The computing device 30 may also communicate with one or more networks via the network interface 16. The network interface 16 may be a wireless network interface.

[0122] The bus 22 may include an address bus, a data bus, a control bus, etc. The bus 22 provides a path for exchanging information between components.

[0123] It should be noted that, in the specific implementation process, the computing device 30 may also include other components necessary for normal operation. In addition, those skilled in the art may understand that the above device may only include components necessary for implementing the embodiments of this specification, and need not include all components shown in the figure.

[0124] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), a network storage device, a cloud storage device, or any type of medium or device suitable for storing instructions and / or data.

[0125] An embodiment of the present invention also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0126] Those skilled in the art can clearly understand that the technical solution of the present invention can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array, an integrated circuit, etc.

[0127] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0128] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended clauses.

Claims

1. A method for identifying and alarming safety helmets in mines, characterized in that: include: Acquire a frame image of a real-time video stream from a camera device, wherein the real-time video stream is used for fixed monitoring of a workplace in a mine; Performing human body detection and recognition on the frame image to obtain a human body recognition area; Positioning the head region of the human body recognition area to obtain the head recognition area; Performing a helmet wearing recognition detection on the head recognition area to obtain a helmet wearing recognition result; The helmet wearing recognition result is reviewed based on the semantic segmentation model, and it is determined whether to output a helmet not wearing alarm signal according to the review result.

2. The method according to claim 1, characterized in that Performing human body detection and recognition on the frame image to obtain a human body recognition area includes: Performing human body detection and recognition on the frame image by using a human body detector to obtain a human body target frame and a detection score for each person in the frame image; If the detection score is greater than a set first threshold, the corresponding human target frame is determined to be valid, and the human target frame is used as the human recognition area for further recognition.

3. The method according to claim 2, characterized in that Before performing human body detection and recognition on the frame image to obtain the human body recognition area, the method further includes: Adjusting the input size of the frame image; The human detector is trained using a mine pedestrian dataset.

4. The method according to claim 2, characterized in that: Positioning the head region of the human body recognition area to obtain the head recognition area includes: The head region of the human body recognition area is located by a head and shoulder key point detector network to obtain the head and shoulder key points and the visible attributes, key point coordinates and corresponding confidence scores of the head and shoulder key points, wherein the head and shoulder key points include: head vertex, neck point, right shoulder point and left shoulder point; If the confidence score of the key point is greater than a second threshold, the head recognition area is obtained according to the coordinates of the head and shoulder key points.

5. The method according to claim 4, characterized in that Performing helmet wearing recognition detection on the head recognition area to obtain a helmet wearing recognition result includes: If the head vertex visibility attribute of the head recognition area is less than a third threshold, performing a helmet wearing recognition detection on the head recognition area; Performing a helmet wearing recognition detection on the head recognition area to obtain a first probability score of not wearing a helmet; If the first probability score is greater than a fourth threshold, it is preliminarily determined that the person is not wearing a helmet.

6. The method according to claim 5, characterized in that The human detector adopts the yolov7 target detection framework, and the convolution module of the target detection framework adopts the leakyRelu activation layer; the head and shoulder key point detector network is constructed by combining the HRNet network with the ResNet18 with a residual structure; the helmet wearing recognition detection adopts the ResNet50 network structure with a residual structure.

7. The method according to claim 6, characterized in that The helmet wearing recognition result is reviewed based on the semantic segmentation model to determine whether the helmet wearing recognition result is accurate, including: Inputting the head recognition area determined as not wearing a helmet into the semantic segmentation model, wherein the segmentation categories of the semantic segmentation model include: skin, hair, background, and helmet; When the skin contour and hair contour are segmented, and the helmet contour is smaller than the hair contour, it is judged that the helmet is not worn and a warning is issued.

8. A device for identifying and alarming safety helmets in mines, characterized in that: The device comprises: an acquisition module, a human body detection identifier, a head recognition locator, and a helmet-not-wearing detector, wherein: The acquisition module is used to acquire frame images of a real-time video stream from a camera device, and the real-time video stream is used for fixed monitoring of the underground mine workplace; The human body detection and recognition device is used to perform human body detection and recognition on the frame image to obtain a human body recognition area; The head recognition locator locates the head region of the human body recognition area to obtain the head recognition area; The non-helmet-wearing detector performs helmet-wearing recognition detection on the head recognition area to obtain a helmet-wearing recognition result; The region segmentation module reviews the helmet wearing recognition result based on the semantic segmentation model, and determines whether to output a helmet not wearing alarm signal according to the review result.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computing device, characterized in that: include: processor; as well as A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 7.

Images