Safety helmet identification method based on multiple frames

Abstract

The invention relates to the field of multi-frame recognition, in particular to a safety helmet recognition method based on multiple frames. The method comprises the following steps: S1, collecting a video stream through a camera, and obtaining multi-frame images of wearing and non-wearing of the safety helmet from the video stream; s2, training a detection model by using a YOLOv5 network and training a safety helmet classification model by using VGG; s3, loading the trained detection and classification model by using a deep learning target detection framework TensorFlow; s4, inputting a scene image needing to be detected into a deep learning target detection framework TensorFlow to obtain positions corresponding to a human frame and a head frame; s5, performing preliminary classification on the head frame by using the safety helmet classification model; s6, performing matching tracking on the head frame by using a YOLO detection model; s7, counting the proportion of each tracking target in multiple frames or counting the number of continuous frames existing in the target, and giving an identification result if a certain threshold value is exceeded; through the YOLOv5 network training detection model and the VGG training safety helmet classification model, the accuracy of safety helmet detection in a complex environment is improved.

Description

Technical Field

[0001] This invention relates to the field of multi-frame recognition technology, and more specifically, to a method for recognizing safety helmets based on multiple frames. Background Technology

[0002] Typically, deep learning-based object detection is used to detect the positions of person and head bounding boxes within a fixed area, followed by classification using a classification model to identify the safety helmet. However, in complex factory environments, the results are easily affected by environmental factors, leading to numerous false alarms. Therefore, a multi-frame-based safety helmet recognition method is proposed to improve the accuracy of safety helmet recognition. This method first detects the target, then performs matching and tracking to obtain the position of each target in each image. The proportion of tracked targets or the number of continuous tracking frames are then statistically analyzed to identify the safety helmet. Therefore, a multi-frame-based safety helmet recognition method is provided. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-frame-based helmet recognition method to solve the problems mentioned in the background art, such as false alarms caused by complex environmental factors and low accuracy of helmet recognition.

[0004] To achieve the above objectives, the present invention aims to provide a helmet recognition method based on multiple frames, comprising the following steps:

[0005] S1. The camera video stream acquires multiple frames of images showing helmets worn and unworn;

[0006] S2. Use YOLOv5 network to train the detection model and VGG to train the helmet classification model;

[0007] YOLOv5 is an object detection model that uses CSP Deckernet53 as its backbone network and incorporates the PANet structure to improve the efficiency of feature reuse and the ability to detect small objects. Its features include: input layer, feature extraction network, downsampling layer, detection head, anchor box, activation function, and loss function.

[0008] VGG is a deep convolutional neural network model with a simple and unified architecture, consisting of multiple small-sized convolutional kernels and pooling layers. It has a deep network structure and mainly has two architectures: VGG16 and VGG19. VGG16 consists of 16 convolutional layers and VGG19 consists of 19 convolutional layers.

[0009] S3. Load the trained detection and classification models using the deep learning object detection framework TensorFlow;

[0010] TensorFlow achieves the detection and localization of target objects by constructing a convolutional neural network model, which typically consists of multiple convolutional layers, pooling layers, activation functions, and fully connected layers, and uses a large amount of labeled training data for simulated training.

[0011] S4. Input the scene image to be detected into the deep learning object detection framework TensorFlow, use the SSD model to detect the person bounding box and head bounding box and obtain the position of the corresponding person bounding box and head bounding box;

[0012] The core idea of ​​the SSD model is to generate, classify, and regress PriorBoxes on feature maps of different depths, and then fuse them to obtain the final prediction result. It makes predictions on feature maps of different depths, thus enabling the detection of targets of different sizes. It uses prior boxes as the basis for detection; these boxes are predefined on the feature maps, reducing the search space and accelerating the detection process. It directly uses convolutions to extract detection results from different feature maps, rather than using fully connected layers like YOLO. It also performs ample data augmentation, including optical and geometric transformations, providing sufficient training data for the network and effectively improving detection accuracy.

[0013] S5. Use the helmet classification model to perform preliminary classification of the head frame. The helmet classification model is based on the ImageNet1000 classification dataset.

[0014] The ImageNet dataset is a large image dataset established to promote the development of computer image recognition technology. ImageNet contains data for evaluation of classification, localization, and detection tasks. Similar to the classification data, the localization task has 1000 categories, including data on wearing and not wearing helmets (more than 2000 images per category);

[0015] S6. Use the YOLO detection model to match and track the head bounding box. The YOLO detection model is based on the DeepSORT tracking algorithm to achieve more accurate, stable and real-time multi-target tracking.

[0016] S7. Calculate the proportion A of each tracked target in multiple frames. If it exceeds a certain threshold B, output the recognition result; calculate the number of consecutive frames in which the target exists, and if it exceeds a certain threshold b, output the recognition result.

[0017] As a further improvement to this technical solution, step S2 involves training a detection model using a YOLOv5 network and a classification model for the safety helmet using VGG, and includes the following steps:

[0018] S2.1, Collect the image set;

[0019] S2.2 Manually use rectangles to mark the person frame and head frame in the image, with red representing the person frame and green representing the head frame;

[0020] S2.3 Process labeled data and train a model for detecting human bounding boxes and head bounding boxes using the yolov5 network in the deep learning object detection framework TensorFlow;

[0021] S2.4 Collect head frame image data, including data on helmets of various colors and their wearing styles, with more than 2000 images for each category, and train a helmet classification model using VGG.

[0022] As a further improvement to this technical solution, step S2.4 involves training a helmet classification model using VGG, which includes the following steps:

[0023] S3.1 Prepare the training dataset and perform appropriate preprocessing;

[0024] S3.2, Use a pre-trained model;

[0025] S3.3 Remove the existing fully connected layer and add a new fully connected layer to match the number of categories in the classification task;

[0026] S3.4 Training the model with the training dataset, selecting the optimizer and learning rate scheduling strategy.

[0027] As a further improvement to this technical solution, in step S3, the deep learning object detection framework TensorFlow loads the cone model, involving the following steps:

[0028] S4.1 Collect and label datasets containing cone models; the dataset should contain images of cone models as well as the location and category information of each cone model. The quality of the dataset has a significant impact on model performance, and it is necessary to ensure that the dataset has sufficient diversity and representativeness;

[0029] S4.2. Train the object detection framework TensorFlow using the dataset. During training, you can adjust the hyperparameters and model structure as needed to obtain better performance. After training, the model will learn to recognize and locate cone patterns in images.

[0030] S4.3 Deploy the trained model into the safety helmet recognition system to perform real-time recognition of cone-shaped models.

[0031] As a further improvement to this technical solution, in step S4, the input of the scene image to be detected into the target detection framework TensorFlow for detecting person and head bounding boxes using the SSD model involves the following steps:

[0032] S5.1 Training the SSD model, including data preprocessing, model configuration, training, and evaluation;

[0033] S5.2 After training is complete, load the model in the TensorFlow framework for inference;

[0034] S5.3 During the inference phase, the input image is passed to the model, and the model returns the detected person bounding box, head bounding box, and their corresponding confidence score and category label;

[0035] S5.4 The detection results returned by the model need to be post-processed to remove overlapping detection boxes by non-maximum suppression and retain the boxes with the highest confidence.

[0036] S5.5 Visualize the detection results on the original image.

[0037] As a further improvement to this technical solution, in step S4, inputting the scene image to be detected into the object detection framework TensorFlow and using the SSD model to obtain the positions of the person bounding box and the head bounding box involves the following steps:

[0038] S6.1. Multiple convolutional layers are used to extract features at different scales, forming a multi-scale feature pyramid. These feature maps are obtained through different convolutional and pooling layers, ultimately resulting in a series of feature maps with gradually decreasing resolution and sensitivity.

[0039] S6.2. Preset multiple anchor boxes of different sizes and proportions at each feature map location to cover targets of different sizes and shapes; on each feature map, select 4 to 6 anchor boxes of different sizes according to the size and proportion of the anchor boxes; all anchor boxes on each feature map form an anchor box cluster.

[0040] S6.3. On the anchor box cluster, detect each anchor box and output its corresponding category and location information. The classification task uses a softmax layer to compare each anchor box with the categories of K candidate target objects to find the most likely classification result. The regression task uses the SmoothL1 loss function to adjust the position of each anchor box to the optimal state, thereby more accurately enclosing the target;

[0041] The specific calculation process of the SmoothL1 loss function is as follows:

[0042] S6.31 Calculate the difference between the predicted value and the actual value;

[0043] S6.32. Determine whether the absolute value of the difference is less than or equal to a preset threshold, which is usually set to 1;

[0044] S6.32. If the value is greater than the threshold, the loss value is calculated using the Smooth L1 loss function formula.

[0045] The formula for the Smooth L1 loss function is:

[0046]

[0047] Where x represents the difference between the predicted value and the true value; when the absolute value of this difference is less than 1, the SmoothL1 loss function behaves like the L2 loss, which is the square of the difference between the predicted value and the true value; when the absolute value of the difference is greater than or equal to 1, it behaves like the L1 loss, which is the absolute value of the difference between the predicted value and the true value minus 0.5.

[0048] S6.4 Since the same target may be covered by multiple smaller or larger anchor boxes, non-maximum suppression algorithm is needed to eliminate overlapping detection results in each anchor box cluster of each feature map, and only retain the target box with the highest confidence.

[0049] S6.5 Detect the positions of the person bounding box and the head bounding box, and return the corresponding bounding box coordinates and confidence score.

[0050] S6.6 The final retained boxes are the detection results after processing by the non-maximum suppression algorithm. These boxes represent the most likely independent targets in each category.

[0051] Non-maximum suppression algorithms can reduce redundant detections, improve accuracy, increase computational efficiency, enhance visual effects, optimize resource allocation, and improve model robustness. By eliminating overlapping detection boxes, they not only improve detection accuracy and real-time performance but also optimize resource utilization, making the target detection system more practical and efficient in real-world applications.

[0052] As a further improvement to this technical solution, in step S5, a helmet classification model is used to perform preliminary classification of the head frame. This helmet classification model is based on the ImageNet 1000 classification dataset and involves the following steps:

[0053] S7.1 Clearly define the categories of wearing and not wearing safety helmets;

[0054] S7.2 Extract at least 2000 images for each class from the ImageNet 1000 classification dataset for testing;

[0055] S7.3 Load the images to be classified and preprocess them to enhance image quality and features;

[0056] S7.4 Select the ResNet model architecture, input the preprocessed image into the model, and obtain the model's prediction output.

[0057] The ResNet model architecture addresses the vanishing gradient problem in deep neural network training by introducing residual blocks. Each residual block contains two convolutional layers and one skip connection, enabling the network to learn a residual function, which is the difference between the input and output. ResNet's unique structure allows the network to stack deeper layers, thereby improving the network's expressive power and performance.

[0058] As a further improvement to this technical solution, in step S6, the YOLO detection model is used to match and track the head bounding box. The YOLO detection model is based on the DeepSORT tracking algorithm to achieve more accurate, stable, and real-time multi-target tracking. Specifically, it involves the following steps:

[0059] S8.1 Constructing the YOLO detection model;

[0060] The YOLO detection model treats object detection as a regression problem, directly predicting bounding boxes and class probabilities from the complete image;

[0061] S8.2 Extract features from the detected target region;

[0062] S8.3 When a new target is detected for the first time, create a new trajectory for it and use its current frame's bounding box and feature vector as the initial state;

[0063] S8.4 Perform data association by using the Hungarian algorithm to match the target detected in the current frame with the existing trajectory; set a threshold P, and only when the similarity between two features exceeds the threshold are they considered to be different observations of the same target;

[0064] The Hungarian algorithm is used to solve the allocation problem. It has low time complexity, simple implementation, and strong practicality, and is a classic combinatorial optimization algorithm.

[0065] S8.5 For each confirmed trajectory, a Kalman filter is used to predict the possible position of the target in the next frame and its state is updated; if the predicted position is close to the actual detected position, the continuity of the trajectory is further confirmed.

[0066] The Kalman filter is a recursive filtering algorithm mainly used to solve the problem of how to estimate the system state from a series of incomplete and inaccurate measurement data in a noisy dynamic environment. In the estimation process, it takes into account the dynamic model of the system (i.e. how the system evolves over time) and the observation data, and can give the optimal estimate of the system state at each time step. Here, optimal means optimal in the sense of minimum mean square error.

[0067] S8.6 Newly detected targets are initially marked as "unconfirmed" and upgraded to "confirmed" after several consecutive frames of matching; trajectories that have not been matched for a long time should be marked as ended and removed from the list of active trajectories;

[0068] S8.7 Repeat the above process for each frame in the video sequence to continuously update and manage the trajectory of the target.

[0069] As a further improvement to this technical solution, step S7, which involves calculating the proportion of each tracked target across multiple frames, includes the following steps:

[0070] S9.1 Detect the tracking target in each frame of the image and set the threshold B;

[0071] S9.2 The detected tracking target needs to be associated with the target in the previous frame to form a continuous trajectory; a unique ID is assigned to the successfully matched target;

[0072] S9.3 Use a list to record the number of frames in which each target ID appears. For each frame, iterate through all tracked targets and increment the frame count for the corresponding ID. At the end of the entire video that needs to be counted, summarize the frame count for each target ID.

[0073] S9.4 Calculate the frame percentage A for each target ID. If the threshold is set to B, the recognition result will be output when the number of consecutive frames exceeds a certain threshold B.

[0074] As a further improvement to this technical solution, in step S7, counting the number of consecutive frames in which the target exists involves the following steps:

[0075] S10.1 Initialize the data structure to store each target ID and its corresponding consecutive frame information; a dictionary can be used, where the key is the target ID and the value is a list used to record the frame number in which the target appears, and a threshold b is set;

[0076] S10.2 Perform object detection on each frame of the video to obtain the bounding box and features of the object; for newly detected objects, assign a new object ID and add it to the data structure, with an initial frame number of 1;

[0077] S10.3 Match the target in the current frame with the target in the previous frame. For a successfully matched target, update its frame number list in the data structure and add the current frame number.

[0078] S10.4 For each target, traverse its frame number list in the data structure and calculate the number of consecutive frames. The number of consecutive frames can be determined by comparing the difference between adjacent frame numbers. If the difference is 1, it indicates that the frames are consecutive; otherwise, it indicates that the continuity is interrupted. At this time, the end point of the consecutive frame count can be recorded.

[0079] S10.5 Count the number of consecutive frames where the target exists, a. If a threshold value is set to b, the recognition result will be output when the number of consecutive frames exceeds a certain threshold b.

[0080] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0081] 1. In this multi-frame-based helmet recognition method, a detection model is trained using a YOLOv5 network and a helmet classification model is trained using VGG. Efficiency is emphasized, providing fast training and inference speeds while maintaining high accuracy. CSPDarknet53 is used as the backbone network, combined with a PANet structure to improve feature reuse efficiency and the ability to detect small targets. The VGG network demonstrates the importance of depth in improving image classification accuracy by using deeper convolutional layers.

[0082] 2. In this multi-frame-based helmet recognition method, the helmet classification model is used to perform preliminary classification of the head frame. The helmet classification model is based on the ImageNet 1000 classification dataset. The ImageNet dataset is a large image dataset established to promote the development of computer image recognition technology. ImageNet has data similar to the classification data for evaluation of classification, localization and detection tasks. The localization task has 1000 categories, including data of wearing helmets and not wearing helmets (more than 2000 images per category), which can better identify whether a helmet is worn and improve the accuracy of helmet recognition. Attached Figure Description

[0083] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] Example:

[0086] Please see Figure 1 As shown, this embodiment provides a helmet recognition method based on multiple frames, including the following steps:

[0087] S1. The camera video stream acquires multiple frames of images showing helmets worn and unworn;

[0088] S2. Use YOLOv5 network to train the detection model and VGG to train the helmet classification model;

[0089] YOLOv5 is an object detection model that uses CSPDarknet53 as its backbone network and incorporates the PANet structure to improve the efficiency of feature reuse and the ability to detect small objects. Its features include: input layer, feature extraction network, downsampling layer, detection head, anchor box, activation function, and loss function.

[0090] VGG is a deep convolutional neural network model with a simple and unified architecture, consisting of multiple small-sized convolutional kernels and pooling layers. It has a deep network structure and mainly has two architectures: VGG1 6 and VGG1 9. VGG1 6 consists of 16 convolutional layers, while VGG1 9 consists of 19 convolutional layers.

[0091] In this embodiment, the YOLOv5 network is used to train the detection model and the VGG network is used to train the helmet classification model, involving the following steps:

[0092] S2.1, Collect the image set;

[0093] S2.2 Manually use rectangles to mark the person frame and head frame in the image, with red representing the person frame and green representing the head frame;

[0094] S2.3 Process labeled data and train a model for detecting human bounding boxes and head bounding boxes using the yolov5 network in the deep learning object detection framework TensorFlow;

[0095] S2.4 Collect head frame image data, including data on helmets and helmet wearing for each color, with more than 2000 images for each category. Train a helmet classification model using VGG, which is based on the ImageNet1000 classification dataset.

[0096] Furthermore, a classification model based on the ImageNet 1000 classification dataset is trained using VGG. First, the VGG model is pre-trained on the ImageNet dataset to learn general image features, involving the following steps:

[0097] S3.1 Prepare the training dataset and perform appropriate preprocessing; resize images and augment data to increase the model's generalization ability;

[0098] S3.2, Use a pre-trained model;

[0099] S3.3 Remove the existing fully connected layer and add a new fully connected layer to match the number of categories in the classification task;

[0100] S3.4 Training the model with the training dataset, selecting the optimizer and learning rate scheduling strategy;

[0101] S3. Load the trained detection and classification models using the deep learning object detection framework TensorFlow;

[0102] In this embodiment, TensorFlow constructs a convolutional neural network model, which typically consists of multiple convolutional layers, pooling layers, activation functions, and fully connected layers. It uses a large amount of labeled training data for simulated training to achieve the detection and localization of target objects.

[0103] Loading a cone model using the deep learning object detection framework TensorFlow involves the following steps:

[0104] S4.1 Collect and label datasets containing cone models; the dataset should contain images of cone models as well as the location and category information of each cone model. The quality of the dataset has a significant impact on model performance, and it is necessary to ensure that the dataset has sufficient diversity and representativeness;

[0105] S4.2. Train the object detection framework TensorFlow using the dataset. During training, you can adjust the hyperparameters and model structure as needed to obtain better performance. After training, the model will learn to recognize and locate cone patterns in images.

[0106] S4.3 Deploy the trained model into the safety helmet recognition system to perform real-time recognition of cone-shaped models.

[0107] S4. Input the scene image to be detected into the deep learning object detection framework TensorFlow, use the SSD model to detect the person bounding box and head bounding box and obtain the position of the corresponding person bounding box and head bounding box;

[0108] The core idea of ​​the SSD model is to generate, classify, and regress PriorBoxes on feature maps of different depths, and then fuse them to obtain the final prediction result. It makes predictions on feature maps of different depths, thus enabling the detection of targets of different sizes. It uses prior boxes as the basis for detection; these boxes are predefined on the feature maps, reducing the search space and accelerating the detection process. It directly uses convolutions to extract detection results from different feature maps, rather than using fully connected layers like YOLO. It also performs ample data augmentation, including optical and geometric transformations, providing sufficient training data for the network and effectively improving detection accuracy.

[0109] In this embodiment, the process of inputting the scene image to be detected into the TensorFlow object detection framework and using the SSD model to detect person and head bounding boxes involves the following steps:

[0110] S5.1 Training the SSD model, including data preprocessing, model configuration, training, and evaluation;

[0111] S5.2 After training is complete, load the model in the TensorFlow framework for inference;

[0112] S5.3 During the inference phase, the input image is passed to the model, and the model returns the detected person bounding box, head bounding box, and their corresponding confidence score and category label;

[0113] S5.4 The detection results returned by the model need to be post-processed to remove overlapping detection boxes by non-maximum suppression and retain the boxes with the highest confidence.

[0114] S5.5 Visualize the detection results on the original image.

[0115] The process of inputting the scene image to be detected into the TensorFlow object detection framework and using the SSD model to obtain the positions of the person bounding box and head bounding box involves the following steps:

[0116] S6.1. Multiple convolutional layers are used to extract features at different scales, forming a multi-scale feature pyramid. These feature maps are obtained through different convolutional and pooling layers, ultimately resulting in a series of feature maps with gradually decreasing resolution and sensitivity.

[0117] S6.2. Preset multiple anchor boxes of different sizes and proportions at each feature map location to cover targets of different sizes and shapes; on each feature map, select 4 to 6 anchor boxes of different sizes according to the size and proportion of the anchor boxes; all anchor boxes on each feature map form an anchor box cluster.

[0118] S6.3. On the anchor box cluster, detect each anchor box and output its corresponding category and location information. The classification task uses a softmax layer to compare each anchor box with the categories of K candidate target objects to find the most likely classification result. The regression task uses the Smooth L1 loss function to adjust the position of each anchor box to the optimal state, thereby more accurately enclosing the target;

[0119] The specific calculation process for the Smooth L1 loss function is as follows:

[0120] S6.31 Calculate the difference between the predicted value and the actual value;

[0121] S6.32. Determine whether the absolute value of the difference is less than or equal to a preset threshold, which is usually set to 1:

[0122] S6.32. If the value is greater than the threshold, the loss value is calculated using the SmoothL1 loss function formula.

[0123] The formula for the Smooth L1 loss function in its S6.32 is as follows:

[0124]

[0125] Where x represents the difference between the predicted value and the true value; when the absolute value of this difference is less than 1, the SmoothL1 loss function behaves like the L2 loss, which is the square of the difference between the predicted value and the true value; when the absolute value of the difference is greater than or equal to 1, it behaves like the L1 loss, which is the absolute value of the difference between the predicted value and the true value minus 0.5.

[0126] S6.4 Since the same target may be covered by multiple smaller or larger anchor boxes, non-maximum suppression algorithm is needed to eliminate overlapping detection results in each anchor box cluster of each feature map, and only retain the target box with the highest confidence.

[0127] Non-maximum suppression algorithm can reduce redundant detection, improve accuracy, increase computational efficiency, enhance visual effects, optimize resource allocation, and improve model robustness. By eliminating overlapping detection boxes, it not only improves the accuracy and real-time performance of detection, but also optimizes resource usage, making the target detection system more practical and efficient in real-world applications.

[0128] The non-maximum suppression algorithm can remove a large number of overlapping predicted bounding boxes and retain the bounding boxes most likely to represent the target. The specific process is as follows:

[0129] S6.41. Obtain all detected target candidate boxes. Each candidate box typically contains location information, category prediction, and confidence score. Set an intersection-union threshold H to determine whether two boxes highly overlap.

[0130] S6.42. Sort the candidate boxes in descending order of confidence score; sort them separately for each category, if the model is multi-class.

[0131] S6.43. Select the candidate box with the highest confidence from the sorted list and use it as the initial detection result to be retained;

[0132] S6.44. For the remaining candidate boxes, calculate their intersection-union ratio (IUR) with the currently retained boxes. If the IUR of a candidate box with a retained box is greater than the set threshold A, then the two boxes are considered to represent the same object, and the candidate box is removed from the candidate list to avoid duplicate detection.

[0133] S6.45. The final retained boxes are the detection results after processing by the non-maximum suppression algorithm. These boxes represent the most likely independent targets in each category.

[0134] S6.5 Detect the positions of the person bounding box and the head bounding box, and return the corresponding bounding box coordinates and confidence score.

[0135] S5. Use the helmet classification model to perform preliminary classification of the head frame, where the helmet classification model is based on the ImageNet 1000 classification dataset;

[0136] The ImageNet dataset is a large image dataset established to promote the development of computer image recognition technology. ImageNet contains data for evaluation of classification, localization, and detection tasks. Similar to the classification data, the localization task has 1000 categories;

[0137] Furthermore, a helmet classification model is used to perform preliminary classification of the head frame. This helmet classification model is based on the ImageNet 1000 classification dataset and involves the following steps:

[0138] S7.1 Clearly define the categories of wearing and not wearing safety helmets;

[0139] S7.2 Extract at least 2000 images for each class from the ImageNet 1000 classification dataset for testing;

[0140] S7.3 Load the images to be classified and preprocess them to enhance image quality and features;

[0141] S7.4 Select the ResNet model architecture, input the preprocessed image into the model, and obtain the model's prediction output.

[0142] The ResNet model architecture addresses the vanishing gradient problem in deep neural network training by introducing residual blocks. Each residual block contains two convolutional layers and one skip connection, enabling the network to learn a residual function, which is the difference between the input and output. ResNet's unique structure allows the network to stack deeper layers, thereby improving the network's expressive power and performance.

[0143] S6. Use the YOLO detection model to match and track the head bounding box. The YOLO detection model is based on the DeepSORT tracking algorithm to achieve more accurate, stable, and real-time multi-target tracking. The specific steps are as follows:

[0144] S8.1 Constructing the YOLO detection model;

[0145] The YOLO detection model treats object detection as a regression problem, directly predicting bounding boxes and class probabilities from the complete image;

[0146] S8.2 Extract features from the detected target region;

[0147] S8.3 When a new target is detected for the first time, create a new trajectory for it and use its current frame's bounding box and feature vector as the initial state;

[0148] S8.4 Perform data association by using the Hungarian algorithm to match the target detected in the current frame with the existing trajectory; set a threshold P, and only when the similarity between two features exceeds the threshold are they considered to be different observations of the same target;

[0149] The Hungarian algorithm, used to solve allocation problems, has low time complexity, is simple to implement, and is highly practical. It is a classic combinatorial optimization algorithm, specifically:

[0150] S8.41. Establish an auxiliary matrix to represent the connection relationship between set X and set Y in a bipartite graph. The elements in the matrix can be the weights of the edges.

[0151] S8.42. For each row of the auxiliary matrix, find the minimum value in that row, and then subtract this minimum value from all elements in that row. Then, do the same operation for each column of the modified matrix, find the minimum value and subtract it. This process may need to be repeated many times until there are no negative numbers in the matrix. At this point, all 0s represent potential matching relationships.

[0152] S8.43. Starting from any unmatched vertex in the X set, try to move to the Y set through unmatched edges (represented as 0 in the matrix) and then return to the X set along matched edges. This process is to find an augmenting path, that is, starting from an unmatched X vertex, alternately passing through unmatched and matched edges, and finally reaching another unmatched X vertex or forming a loop back to the starting point.

[0153] S8.44 If an augmenting path is found, update the matching relationship along the path: originally matched edges are unmatched, and unmatched edges on the path become matched; this is equivalent to adding an edge to the matching set, and may also release other vertices, making them new unmatched vertices;

[0154] S8.45. Repeat the search for augmenting paths until no more augmenting paths can be found; the matching set of edges is the result of maximizing the matching.

[0155] S8.5 For each confirmed trajectory, use a Kalman filter to predict the possible position of the target in the next frame and update its state; if the predicted position is close to the actual detected position, further confirm the continuity of the trajectory.

[0156] The Kalman filter is a recursive filtering algorithm primarily used to solve the problem of estimating the system state from a series of incompletely accurate measurement data in noisy dynamic environments. During the estimation process, it considers the system's dynamic model (i.e., how the system evolves over time) and the observed data, providing an optimal estimate of the system state at each time step. Here, "optimal" refers to the optimal estimate in the sense of minimum mean square error. Specifically:

[0157] S8.51. Set the initial state estimate and the initial state covariance matrix; these parameters are usually determined based on the target's initial position and velocity.

[0158] S8.52. At each time step, the Kalman filter first predicts the system state based on the state equation; the predicted value is calculated based on the state estimate of the previous time step and the system noise covariance matrix.

[0159] Among them, state prediction Predicting the next state using a system model typically involves the state transition matrix A and the possible control inputs B·u. k ,Right now

[0160]

[0161] Covariance prediction (P) k|k-1 Similarly, to predict the covariance matrix, considering the effect of process noise, the formula is:

[0162] P k|k-1 =A·P k-1|k-1 ·A T +Q

[0163] Where Q is the covariance matrix of the process noise;

[0164] S8.53. The Kalman filter updates the system state estimate based on the observed values ​​and the observation noise covariance matrix; the updated value is calculated based on the predicted values ​​and the observed values.

[0165] Among them, the Kalman gain (K) is calculated. k () is an important weight in the relationship between the predicted state and the observed value, determining the magnitude of the update; the calculation formula is:

[0166] K k =P k|k-1 ·H T (H·P k|k-1 ·H T +R) -1

[0167] Where H is the observation matrix and R is the covariance matrix of the observation noise;

[0168] S8.54 The Kalman filter repeats the prediction and update steps until convergence or the maximum number of iterations is reached.

[0169] S8.6 Newly detected targets are initially marked as "unconfirmed" and upgraded to "confirmed" after several consecutive frames of matching; trajectories that have not been matched for a long time should be marked as ended and removed from the list of active trajectories;

[0170] S8.7 Repeat the above process for each frame in the video sequence to continuously update and manage the trajectory of the target.

[0171] S7. Calculate the proportion A of each tracked target in multiple frames. If it exceeds a certain threshold B, output the recognition result; calculate the number of consecutive frames in which the target exists, and if it exceeds a certain threshold b, output the recognition result.

[0172] In this embodiment, calculating the proportion of each tracked target across multiple frames involves the following steps:

[0173] S9.1 Detect the tracking target in each frame of the image and set the threshold B;

[0174] S9.2 The detected tracking target needs to be associated with the target in the previous frame to form a continuous trajectory; a unique ID is assigned to the successfully matched target to ensure the consistency of tracking across frames;

[0175] S9.3 Use a list to record the number of frames in which each target ID appears. For each frame, iterate through all tracked targets and increment the frame count of the corresponding ID. At the end of the entire video that needs to be counted, summarize the frame count of each target ID.

[0176] S9.4 Calculate the frame percentage A of each target ID, which is the number of frames the target is tracked divided by the total number of frames. If it exceeds a certain threshold B, output the recognition result.

[0177] Furthermore, determining the number of consecutive frames in which the target exists involves the following steps:

[0178] S10.1 Initialize the data structure to store each target ID and its corresponding consecutive frame information; a dictionary can be used, where the key is the target ID and the value is a list used to record the frame number in which the target appears, and a threshold b is set;

[0179] S10.2 Perform object detection on each frame of the video to obtain the bounding box and features of the object; for newly detected objects, assign a new object ID and add it to the data structure, with an initial frame number of 1;

[0180] S10.3 Match the target in the current frame with the target in the previous frame. For a successfully matched target, update its frame number list in the data structure and add the current frame number.

[0181] S10.4 For each target, iterate through its frame number list in the data structure and calculate the number of consecutive frames. The number of consecutive frames can be determined by comparing the difference between adjacent frame numbers. A difference of 1 indicates continuity, otherwise it indicates a break in continuity. In this case, the end point of the consecutive frame count can be recorded.

[0182] S10.5. The number of consecutive frames in which the target exists is denoted as 'a'. If a threshold value of 'b' is set, the recognition result will be output when the number of consecutive frames exceeds a certain threshold 'b'.

[0183] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A helmet recognition method based on multiple frames, characterized in that: Includes the following steps: S1. Capture video streams using a camera, and obtain multiple frames of images from the video stream showing helmets being worn and not worn. S2. Train the detection model using the YOLOv5 network and the helmet classification model using VGG; S3. Load the trained detection and classification models using the deep learning object detection framework TensorFlow; S4. Input the scene image to be detected into the deep learning object detection framework TensorFlow, use the SSD model to detect the person bounding box and head bounding box and obtain the position of the corresponding person bounding box and head bounding box; S5. Use a helmet classification model to perform preliminary classification of the head frame to identify whether a helmet is being worn. S6. Use the YOLO detection model to match and track the head frame; S7. Calculate the proportion of each tracked target in multiple frames as A. If the threshold value is set to B, the recognition result will be output when the proportion of each tracked target exceeds a certain threshold B; or calculate the number of consecutive frames in which the target exists as a. If the threshold value is set to b, the recognition result will be output when the number of consecutive frames exceeds a certain threshold b.

2. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S2, the detection model is trained using the YOLOv5 network and the helmet classification model is trained using VGG, involving the following steps: S2.1, Collect the image set; S2.2 Manually use rectangles to mark the person frame and head frame in the image, with red representing the person frame and green representing the head frame; S2.3 Process labeled data and train a model for detecting human bounding boxes and head bounding boxes using the YOLOv5 network in the deep learning object detection framework TensorFlow; S2.4 Collect head frame thumbnail data and use VGG to train a helmet classification model.

3. The helmet recognition method based on multiple frames according to claim 2, characterized in that: In step S2.4, training the helmet classification model using VGG involves the following steps: S3.1 Prepare the training dataset and perform appropriate preprocessing; S3.2, Use a pre-trained model; S3.3 Remove the existing fully connected layer and add a new fully connected layer to match the number of categories in the classification task; S3.4 Training the model with the training dataset, selecting the optimizer and learning rate scheduling strategy.

4. The helmet recognition method based on multiple frames according to claim 4, characterized in that: In S3, the deep learning object detection framework TensorFlow loads the cone model, involving the following steps: S4.1 Collect and label datasets containing cone models; S4.

2. Train the object detection framework TensorFlow using the dataset; S4.3 Deploy the trained model into the safety helmet recognition system to perform real-time recognition of cone-shaped models.

5. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S4, inputting the scene image to be detected into the TensorFlow object detection framework and using the SSD model to detect person and head bounding boxes involves the following steps: S5.1 Training the SSD model; S5.2 After training is complete, load the model in the TensorFlow framework for inference; S5.3 During the inference phase, the input image is passed to the model, and the model returns the detected person bounding box, head bounding box, and their corresponding confidence score and category label; S5.4 The detection results returned by the model need to be post-processed to remove overlapping detection boxes by non-maximum suppression and retain the boxes with the highest confidence. S5.5 Visualize the detection results on the original image.

6. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S4, inputting the scene image to be detected into the object detection framework TensorFlow and using the SSD model to obtain the positions of the person bounding box and head bounding box involves the following steps: S6.

1. Use multiple convolutional layers to extract features at different scales to form a multi-scale feature pyramid; S6.

2. Preset multiple anchor boxes of different sizes and proportions at each feature map location to cover targets of different sizes and shapes; on each feature map, select 4 to 6 anchor boxes of different sizes according to the size and proportion of the anchor boxes; all anchor boxes on each feature map form an anchor box cluster. S6.3 On the anchor box cluster, detect each anchor box and output its corresponding category and location information; use the softmax layer for classification tasks and the Smooth L1 loss function for regression tasks; S6.4 Use the non-maximum suppression algorithm to eliminate overlapping detection results and retain only the target boxes with the highest confidence. S6.5 Detect the positions of the person bounding box and the head bounding box, and return the corresponding bounding box coordinates and confidence score.

7. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S5, a helmet classification model is used to perform preliminary classification of the head frame. This helmet classification model is based on the ImageNet 1000 classification dataset, and the classification involves the following steps: S7.1 Clearly define the categories of wearing and not wearing safety helmets; S7.2 Extract at least 2000 images for each class from the ImageNet 1000 classification dataset for testing; S7.3 Load the images to be classified and preprocess them; S7.4 Select the ResNet model architecture, input the preprocessed image into the model, and obtain the model's prediction output.

8. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S6, the YOLO detection model is used to match and track the head bounding box. The YOLO detection model is based on the DeepSORT tracking algorithm to achieve more accurate, stable, and real-time multi-target tracking. Specifically, it involves the following steps: S8.1 Constructing the YOLO detection model; S8.2 Extract features from the detected target region; S8.3 When a new target is detected for the first time, create a new trajectory for it and use its current frame's bounding box and feature vector as the initial state; S8.4 Perform data association by matching the target detected in the current frame with the existing trajectory; S8.5 For each confirmed trajectory, predict the possible position of the target in the next frame and update its state; S8.6 Newly detected targets are initially marked as "unconfirmed". After several frames of continuous matching, they are upgraded to "confirmed". Trajectories that have not been matched for a long time should be marked as ended and removed from the list of active trajectories. S8.7 Repeat the above process for each frame in the video sequence to continuously update and manage the trajectory of the target.

9. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S7, calculating the proportion of each tracked target across multiple frames involves the following steps: S9.1 Detect the tracking target in each frame of the image and set the threshold B; S9.2 The detected tracking target needs to be associated with the target in the previous frame to form a continuous trajectory; a unique ID is assigned to the successfully matched target; S9.3 Use a list to record the number of frames in which each target ID appears. For each frame, iterate through all tracked targets and increment the frame count of the corresponding ID. At the end of the entire video that needs to be counted, summarize the frame count of each target ID. S9.4 Calculate the frame percentage A for each target ID. If the threshold is set to B, the recognition result will be output when the number of consecutive frames exceeds a certain threshold B.

10. The helmet recognition method based on multiple frames according to claim 1, characterized in that: In step S7, counting the number of consecutive frames in which the target exists involves the following steps: S10.1 Initialize the data structure to store each target ID and its corresponding consecutive frame information, and set the threshold b; S10.2 Perform object detection on each frame of the video to obtain the bounding box and features of the object; For a newly detected target, assign a new target ID and add it to the data structure, with an initial frame count of 1; S10.3 Match the target in the current frame with the target in the previous frame. For a successfully matched target, update its frame number list in the data structure and add the current frame number. S10.4 For each target, iterate through its frame number list in the data structure and calculate the number of consecutive frames; S10.

5. The number of consecutive frames in which the target exists is denoted as 'a'. If a threshold value of 'b' is set, the recognition result will be output when the number of consecutive frames exceeds a certain threshold 'b'.

Images