Incremental learning and role determination method for helmet recognition based on deep learning

Abstract

The present invention discloses a method for incremental learning and role determination of helmet recognition based on deep learning. The camera is streamed through RTSP, and then the collected video is frame skipped. The batch information of the training data added to the data set is recorded, and the network is fine-tuned after the new training data is added to achieve incremental learning; the learning rules are determined according to the batches to avoid catastrophic forgetting. The helmets of different colors and the situation of not wearing a helmet in the video are detected and identified. While determining the unsafe behavior of not wearing a helmet, the role of the person wearing the helmet is determined by the color of the helmet. The present invention combines the target detection technology with the safety mechanism of the factory to complete the risk detection and early warning system. While saving a lot of manpower and material resources, it improves the safety protection mechanism of the factory, improves the factory management system, and protects the life, health and safety of the personnel.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and relates to a monitoring video incremental learning method based on deep learning, the recognition of the wearing condition of a helmet, the distinction of the color of the helmet, the judgment of a person's role, and the design of an alarm system. Background Art

[0002] In many factories, wearing a hard hat has become a mandatory requirement. A hard hat can effectively prevent sudden flying objects from hitting the head, prevent electric shocks, and prevent hair from being caught in the machine or exposed to dust. However, its importance is easily overlooked by some workers, resulting in safety accidents.

[0003] Relying on security guards and other staff to manually check the wearing of helmets and identify unsafe behaviors is very unstable and subjective, and it is difficult to monitor in real time without blind spots. At the same time, different colors of helmets represent different roles. In actual scenarios, different jobs require different personnel to perform. At this time, the color of the helmet can be used to correctly determine the role of the person to determine whether certain operations can be performed and prevent the entry and operation of unauthorized personnel. This project relies on human identification and has great limitations and instability.

[0004] At present, deep learning plays an increasingly important role in the field of target detection. It uses advanced convolutional neural theory, deep belief network, neural network, etc. to build a multi-level learning system, which can realize feature detection, target detection, etc., and improve the accuracy, reliability and advancement of machine learning. In particular, the emergence of YOLO v5 has promoted the accuracy and precision of target detection. In the application field of visual algorithms, recognition algorithms often fail due to the emergence of new categories (such as new colors of helmets) and completely different new scenes. If a model is trained from scratch, it is very expensive. To solve this problem, incremental learning is required, and in incremental learning, catastrophic forgetting is the main situation to be avoided. This solution solves this technical problem. Summary of the invention

[0005] The purpose of the present invention is to provide a method for incremental learning and role determination of helmet recognition based on deep learning, which solves the technical problem of how to improve the factory safety protection mechanism. It provides a method that can timely obtain camera video, process the video, identify the wearing status of the staff's helmet, and distinguish the staff's role by the different colors of the helmet, so as to timely detect unsafe behaviors of not wearing a helmet and whether it complies with the safety production regulations. In addition, it can also support incremental learning and avoid catastrophic forgetting.

[0006] The present invention adopts the ideas of incremental learning and role mapping, and proposes a method for incremental learning and role determination of helmet recognition based on deep learning, which can not only perform helmet detection and realize automatic safety supervision, but also realize role recognition and provide technical support for subsequent business. Taking into account the problem of frequent new categories and scenarios in actual factory environments, incremental learning is used to avoid high-cost zero-starting point learning, and batch method is used to avoid catastrophic forgetting in incremental learning. The system includes: training module, camera scheduling module, GPU scheduling module, personnel detection module, role mapping module, risk reporting and recording module; among which:

[0007] Training module: responsible for collecting training data, establishing learning rules, achieving incremental learning, and avoiding catastrophic forgetting.

[0008] Camera scheduling module: Based on RTSP, it realizes the stream acquisition of the working camera, polls the camera and extracts frames, and stores the data in the buffer to be detected.

[0009] The implementation method of the camera scheduling module is as follows:

[0010] Build a global image linked list, and one thread polls all cameras;

[0011] For each camera, obtain the stream based on RTSP and extract the current frame;

[0012] Lock the global image list, store the current frame, and unlock it. The process is as follows:

[0013] ifIs.Length <MaxLen

[0014] {

[0015] Lock(Is);

[0016] Is.Add(I);

[0017] Unlock(Is);

[0018] }

[0019] Where Is is the global image list, I is the current image obtained by extracting frames, and MaxLen is the maximum length allowed by the list. When the list reaches the maximum length, stop extracting images and wait for the images in the list to be consumed by the GPU scheduling module.

[0020] GPU scheduling module: Each GPU is equipped with a controller. Each controller detects the number of images in the buffer to be detected. When the number reaches a certain threshold, it forms a batch for prediction, and then pushes the prediction results back through the queue to improve model efficiency and accuracy.

[0021] The GPU scheduling module is implemented as follows:

[0022] Each GPU scheduling module independently detects whether the length of the global image list exceeds a certain threshold;

[0023] When the detection exceeds the threshold, the global linked list is locked, the threshold number of images are obtained, deleted from the global linked list, and then unlocked;

[0024] The acquired images are grouped into a batch and loaded into the GPU for training.

[0025] The process is as follows:

[0026]

[0027] MinLen is the threshold value of the number of images that the GPU can process in one batch.

[0028] Personnel detection module: Based on the improved YOLO v5 target detection model, detect and identify whether a person is wearing a helmet and the color of the helmet; the implementation method of the personnel detection module is as follows:

[0029] The detection box is calculated as follows:

[0030]

[0031] where x * and x a Represents the x coordinate and y coordinate of the upper left corner of the prediction box and anchor box respectively. a 、w a 、h a They are the y coordinate of the upper left corner, the frame width, and the frame height respectively; These are the box regression parameters obtained during training.

[0032] Detection box category: in To predict the probability of belonging to a certain class, c i is the category label.

[0033] Role mapping module: maps the detected helmet color to the personnel role; the implementation method of the role mapping module is as follows:

[0034] Constructing a color-role mapping function in It is a collection of helmet colors. is a set of roles. After identifying the color of the helmet, the role is determined through the mapping function.

[0035] Risk reporting and recording module: responsible for uploading, storing, and alerting the information and photo evidence of personnel who are not wearing helmets, and providing historical records and evidence query; the implementation method of the risk reporting and recording module is as follows:

[0036] After the personnel detection module detects the information, it sends a Kafka message;

[0037] After receiving the Kafka message, the risk reporting and recording module determines the situation of not wearing a helmet as a risk situation and persists the evidence and other information;

[0038] The risk reporting and recording modules pop up alert information;

[0039] The risk reporting and recording module provides follow-up query functions.

[0040] The specific implementation method of the training module is as follows:

[0041] On the basis of the original training set, we collect pictures of new categories and annotate them. The annotated labels use the following data (I, x, y, w, h, c, b), where I is the ID of the image, x is the x-coordinate of the upper left corner of the annotation box, y is the y-coordinate of the upper left corner of the annotation box, w is the width of the annotation box, h is the height of the annotation box, c is the category label of the annotation box, and b is the batch of annotation boxes.

[0042] Regarding the determination of the batch of annotation boxes, the batch size is set to 1 when the training set is initialized, and the batch size increases by 1 for subsequent newly added images.

[0043] In incremental learning, fine-tuning is performed on the basis of the original training model to prevent catastrophic forgetting through the following loss function:

[0044]

[0045] Among them I obj is an indicator function, indicating that an anchor window is involved in the calculation only when it is responsible for this category; b i is the batch information of the annotation box recorded in step i, c i is the label box category label recorded in step i, p j (c i ) is the predicted probability.

[0046] Regarding the detection box regression, the following coordinate calculation is used:

[0047]

[0048] where x * and x a Represents the x coordinate and y coordinate of the upper left corner of the prediction box and anchor box respectively. a 、w a 、h a They are the y coordinate of the upper left corner, the frame width, and the frame height respectively; These are the box regression parameters obtained during training. The CIOU loss function is used for regression.

[0049] The present invention is divided into multiple parts, each of which has its own configuration instructions. By changing different configuration files, the corresponding configuration can be modified, thereby improving the flexibility and scalability of the system. The system can be personalized using configuration files to meet different product requirements, with future scalability, while improving ease of use and maintainability.

[0050] The beneficial effects of the present invention are:

[0051] The present invention combines deep learning technology with factory safety mechanisms. On the basis of identifying the safety helmet, it further distinguishes the color of the safety helmet, thereby distinguishing the role of the staff, and takes real-time streams from the camera through RTSP, timely monitoring the activity status of staff wearing safety helmets of different colors, and determining whether the staff has unsafe behaviors and risk information. When unsafe behaviors occur, risk pictures are uploaded in time and risk information is issued. When adding new identification types and scenarios, incremental learning can be achieved, while preserving the memory of past learning, ensuring that the invention can operate long-term, safely, stably, reliably, and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a system structure diagram in an embodiment of the present invention.

[0053] Figure 2 It is a flow chart of the training model in an embodiment of the present invention.

[0054] Figure 3 It is a data flow diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly illustrate the technical features of this solution, this solution is described below through specific implementation methods.

[0056] The present invention relates to a method for helmet recognition and role determination based on deep learning. The invention can be mainly divided into two major processes: training and operation.

[0057] See also Figure 1 The training module is responsible for model training, and the obtained data model provides a basis for the intelligent recognition of the personnel detection module. The operation module first uses the camera scheduling module to obtain the frame image and store it in the global image list. The GPU scheduling module continuously detects the image list, and when there is sufficient data, it is packaged into a batch and sent to the personnel detection module for helmet recognition. The recognition result determines the role through the role mapping module. At the same time, the risk information is recorded in the risk reporting and recording module, and the evidence is retained.

[0058] See also Figure 2, obtain the existing video and image data from the original security camera's unit, select the video and image data under different angles, different scenes, different time periods, and when there is a large flow of people wearing hats of various colors, or directly obtain the stream from the camera through the RTSP protocol, and then transmit the obtained video and image data to this system.

[0059] During the video frame extraction process, the system needs to perform frame skipping on the obtained video. In order to be as close to the actual situation of the factory as possible and improve the accuracy of the present invention, frames with the same interval time are selected during frame skipping, and then necessary selection is performed on the obtained images. Some images have unclear personnel changes or no personnel appear in the images, and the personnel in the images are blurred. These images that do not have a positive effect on model training are discarded accordingly. Images with obvious personnel flow, various colors of helmets, and comprehensive close and long-distance shooting of personnel are retained.

[0060] The obtained pictures are annotated, mainly relying on manpower and related annotation software at the same time. Because the acquired data is limited, a series of data enhancement operations are performed on the obtained data to improve the accuracy of the present invention, and then the staff wearing helmets of different colors are annotated, the same color helmets are annotated as the same category, and different colors of helmets are annotated as different categories. The staff without helmets are annotated as the category of not wearing helmets. During the annotation process, some helmets are held in the hands of the staff or held in front of the body, placed behind the back or held on the top of the head, and the helmets have been separated from the head. According to regulations, these situations do not belong to the situation of correctly wearing helmets, so they are all listed as the situation of not wearing helmets.

[0061] After the data is labeled, it is divided into training sets, validation sets, and test sets, and then the model is built for training. Through the aforementioned detection box regression and incremental model training, the final learning model is obtained, which is used by the personnel detection module.

[0062] See also Figure 3 All image data is stored in a global image linked list, which is the storage and access center of the entire data at runtime. The linked list sets a maximum length MaxLen and a minimum length MinLen.

[0063] The camera scheduling module continuously polls the camera, and after extracting a frame, it determines whether the length of the linked list is less than MaxLen. If not, it discards the extracted frame or holds the frame but waits until the length of the linked list is less than MaxLen. If less, it locks the linked list, inserts the image into the linked list, unlocks it, and continues to poll the camera.

[0064] The GPU scheduling module continuously detects the global image linked list to determine whether the length of the linked list is greater than or equal to MinLen. If less than, wait for 5 seconds and continue to detect. If greater than or equal, lock the linked list, obtain the previous MinLen frames, delete these frames from the global image linked list, and unlock. Then package these acquired frames and send them to the GPU for detection.

[0065] After the image detection is completed, it can be identified whether there are unsafe behaviors without wearing safety helmets and whether the different people represented by different colored safety helmets comply with the corresponding number of personnel requirements, so as to avoid the situation where only one person is actually performing the work that requires two or more people. Unsafe situations and existing risks can be detected in time and then reported to the risk system.

[0066] This system will upload the detected risk information, including video and image materials containing risks, to the storage server for easy subsequent search and retention, and upload the corresponding risk information to the risk processing platform, and then process the risk according to the actual situation and make corresponding solutions in a timely manner.

[0067] Technical features not described in the present invention can be achieved through or by adopting existing technologies and will not be described in detail here. Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. Incremental learning and role determination method for helmet recognition based on deep learning, characterized in that: It includes training module, camera scheduling module, GPU scheduling module, personnel detection module, role mapping module, risk reporting and recording module; among them, Training module: responsible for collecting training data, establishing learning rules, achieving incremental learning, and avoiding catastrophic forgetting; Camera scheduling module: Based on RTSP, it realizes the acquisition of the working camera stream, polls the camera and extracts frames, and stores the data in the buffer to be detected. The implementation method is as follows: Build a global image linked list, and one thread polls all cameras; For each camera, obtain the stream based on RTSP and extract the current frame; Lock the global image list, store the current frame in it, and unlock it; GPU scheduling module: Each GPU is equipped with a controller. Each controller detects the number of images in the buffer to be detected. When the number reaches a certain threshold, it forms a batch for prediction, and then pushes the prediction results back through the queue to improve the efficiency and accuracy of the model. The implementation method is as follows: Each GPU scheduling module independently detects whether the length of the global image list exceeds a certain threshold; When the detection exceeds the threshold, the global linked list is locked, the threshold number of images are obtained, deleted from the global linked list, and then unlocked; The acquired images are grouped into a batch and loaded into the GPU for training; Personnel detection module: Based on the improved YOLO v5 target detection model, detect and identify whether a person is wearing a helmet and the color of the helmet; the implementation method of the personnel detection module is as follows: The detection box is calculated as follows: where x * and x a Represents the x coordinate and y coordinate of the upper left corner of the prediction box and anchor box respectively. a 、w a 、h a They are the y coordinate of the upper left corner, the frame width, and the frame height respectively; These are the box regression parameters obtained during training; Detection box category: in To predict the probability of belonging to a certain class, c i is the category label; Role mapping module: maps the detected helmet color to the personnel role; the implementation method of the role mapping module is as follows: Constructing a color-role mapping function in It is a collection of helmet colors. is a set of roles; after identifying the color of the helmet, the role is determined through the mapping function; Risk reporting and recording module: responsible for uploading, storing, and alerting the detected personnel information and photo evidence of not wearing helmets, and providing historical records and evidence query; the implementation method of the risk reporting and recording module is as follows: After the personnel detection module detects the information, it sends a Kafka message; After receiving the Kafka message, the risk reporting and recording module determines the situation of not wearing a helmet as a risk situation and persists the evidence and other information; The risk reporting and recording modules pop up alert information; The risk reporting and recording module provides follow-up query functions.

2. The method for helmet recognition incremental learning and role determination based on deep learning according to claim 1 is characterized in that: The specific implementation method of the training module is as follows: On the basis of the original training set, we collect pictures of new categories and annotate them. The annotated labels use the following data (I, x, y, w, h, c, b), where I is the ID of the image, x is the x-coordinate of the upper left corner of the annotation box, y is the y-coordinate of the upper left corner of the annotation box, w is the width of the annotation box, h is the height of the annotation box, c is the category label of the annotation box, and b is the batch of annotation boxes; Regarding the determination of the batch of annotation boxes, the batch is set to 1 when the training set is initialized, and the batch increases by 1 when additional images are added later; In incremental learning, fine-tuning is performed based on the original training model to prevent catastrophic forgetting through the following loss function: Among them I obj is an indicator function, indicating that an anchor window is involved in the calculation only when it is responsible for this category; b i is the batch information of the annotation box recorded in step i, c i is the label box category label recorded in step i, p j (c i ) is the predicted probability.

Images