Monitoring safety early warning technology based on computer vision

Abstract

Aiming at dangerous behaviors and potential safety hazards of various scenes, the invention provides a monitoring safety early warning technology based on computer vision, and multi-target identification of pedestrians, vehicles, instrument holding, injuries, smoke, fire, fight, unmanned aerial vehicles and the like is completed through target detection, target classification and image processing technologies. The functions of finding abnormal behaviors, giving an alarm to prompt workers and storing dangerous pictures are achieved. For dangerous behaviors, a rapid identification and early warning method based on abnormal pictures of each scene is established, and workers are accurately and efficiently helped to complete safety prevention and control tasks.

Description

Technical Field

[0001] The present application relates to a surveillance safety warning technology based on computer vision, specifically to the use of target detection, pedestrian attribute recognition and image processing technology to achieve safety warnings in a variety of surveillance scenarios, and accurately identify targets such as pedestrians, vehicles, weapons, injuries, smoke, fire, fights, drones, and pedestrian attribute recognition. Background Art

[0002] Using surveillance to quickly detect dangerous behaviors and identify safety hazards is one of the key factors in ensuring urban safety. The surveillance equipment for each road and scene is intricate, and the number of surveillance videos is huge. Naked eye observation is inefficient, intensive, and easy to fatigue. For example, if there are ten cameras monitoring different corners of an area, in order to detect dangerous situations in time, the monitor needs to constantly observe between the ten surveillance screens, which will also affect the worker's vision health. More importantly, manual observation is prone to missed detection, which has a serious impact on the timely detection and elimination of dangers.

[0003] Computer vision technology uses an image sensor that replaces the human eye to obtain the image of an object, and uses computers to simulate human judgment criteria to understand and identify images, so as to analyze images and draw conclusions. It is one of the most important core technologies at the perception level. Computer vision technology simulates biological vision, analyzes, identifies, detects and tracks the data and information in the captured images, and truly understands these images. At present, this technology has been widely used in security, autonomous driving, medical treatment, etc.

[0004] This application combines security warning with computer vision technology and proposes a monitoring security warning technology based on computer vision. By quickly and effectively classifying and detecting targets in complex scenes, rapid response and efficient warning of security scenes are achieved, freeing staff from the huge workload of real-time observation, and shifting the focus of work to focusing on observing the alarm screen issued by the computer. The specific operation process is: first, use a high-definition camera to obtain images of the monitored area, and use image processing technology to extract and preprocess image information. Then, the target detection algorithm is used to identify the extracted image to achieve the positioning and identification of targets such as pedestrians, vehicles, weapons, injuries, smoke, fire, fights, and drones. Then, if there are pedestrians among the detected targets, the pedestrian area is cropped, and the cropped image will be passed to the resnet network for reasoning to identify the attributes of the pedestrians. The identified target detection results and pedestrian attribute results will be displayed in the video in the form of target boxes and text respectively. Finally, the detected target information is passed to the early warning system to achieve immediate response and processing of security incidents. For example, when a fighting crowd is detected in the monitoring area, the system will immediately start the automatic warning process and send an alarm message to the administrator to remind him to pay attention and take necessary measures in time. This can not only improve safety performance, but also avoid problems such as misjudgment and missed alarms, providing more comprehensive and accurate support for supervision work.

[0005] This method is fast, efficient and accurate, and can greatly improve security efficiency and response speed. At the same time, this method can achieve high customization and personalized needs by analyzing and extracting target features in different scenarios. Summary of the invention

[0006] The purpose of this application is to propose a surveillance safety warning technology based on computer vision, which can realize real-time monitoring of pedestrians, vehicles, weapons, injuries, smoke, fire, fights, drones and other targets in the surveillance screen in various scenarios, as well as the attribute recognition of pedestrians.

[0007] In order to achieve the above objectives, this application combines image processing and computer vision technology to propose a target detection algorithm and a target classification algorithm based on computer vision. Figure 1 As shown in the figure, the surveillance video is input and the deep neural network model is used for target detection to identify various categories in the picture. If labels such as fighting, holding weapons, smoke, fire, and injury appear, an alarm prompt is triggered.

[0008] The method of this application mainly consists of 4 modules, such as Figure 2As shown, it is divided into a YOLOv5 model training module, a pedestrian attribute recognition model training module, a YOLOv5 model prediction module, and a pedestrian attribute recognition model prediction module. The following describes the four modules in the method of this application respectively.

[0009] (1) YOLOv5 model training module

[0010] The main function of the YOLOv5 model training module is to use 11,000 images to train a model with high accuracy and strong generalization ability. Before selecting a model, two issues need to be considered. First, for surveillance videos, before triggering an alarm to alert the staff, it is necessary to first identify the dangerous behavior in the video, and then trigger the alarm. If the model is not accurate enough, false alarms and missed alarms will often occur. Second, since most of the scenarios used are real-time monitoring and real-time detection, it is necessary to consider the size of the model, the speed of detection, the performance of the equipment, and the size of the real-time FPS value. Therefore, the selection of the target detection algorithm requires extremely high robustness, accurate detection of various dangerous behaviors, and a faster detection speed, so as to create a more ideal processing environment for subsequent warning prompts. In this application method, we use YOLOv5 as a security warning target detection algorithm. YOLOv5: You Only Look Once (YOLO) is a real-time target detection algorithm based on deep neural networks. It uses the Darknet architecture, including convolutional neural networks, residual blocks, and upsampling layers, and uses lightweight Backbone networks such as CSPNet and EfficientNet to achieve faster reasoning speed and higher accuracy. In addition, YOLOv5 also uses an adaptive training method, which can automatically optimize hyperparameters and network structures according to different data sets and tasks, so it has strong versatility and robustness. YOLOv5 performs target detection and classification simultaneously through a single neural network, and is very fast and real-time. It performs better in detecting small objects and occlusions. It is one of the more popular target detection algorithms.

[0011] The architecture of the YOLOv5 model algorithm used in the dangerous behavior detection feature extraction method of this application is described in detail. The architecture of the model is as follows Figure 3 As shown in the figure, the entire model architecture mainly consists of four parts: input end, Backbone, Neck, and Head.

[0012] In the input part, the image is scaled to the specified size after input, and pixel normalization is performed for network processing; Mosaic data enhancement is used to use four images and splice them in a random scaling, random cropping and random arrangement manner. Each image has its corresponding frame. After splicing the four images, a new image is obtained, and the corresponding frame of this image is also obtained. Then such a new image is passed into the neural network for learning; adaptive anchor frame calculation, YOLOv5 will calculate the prior frame anchor for different data sets before training. First, the width and height of all targets in the data set are obtained, and each image is scaled proportionally to the resize specified size to ensure that the maximum width and height meet the specified size, and then the bboxes are changed from relative coordinates to absolute coordinates, and the bboxes are filtered to retain the bboxes with width and height greater than or equal to two pixels. Use kmeans clustering to obtain n anchors, and finally evaluate.

[0013] In the Backbone part, the image is first sliced, that is, the Focus structure. The specific operation is to get a value for every other pixel in an image to get four images. The four images are complementary, so that the W and H information are concentrated in the channel space, and the input channel is expanded by 4 times. Finally, a double down-sampling feature map without information loss is obtained; then the image features are extracted through the CBL, CSP1_X, and CSP2_X structures. CBL consists of Conv+Bn+Leaky+relu activation functions. The CSP1_x module consists of a series of conventional convolutions, BN layers, and LeakyReLU activation functions, followed by a CSP structure and some additional convolution and pooling layers. CSP2_x adds some additional dilated convolution operations after the CSP structure to further expand the receptive field and deepen the feature map, so as to better capture and represent the target information of complex scenes.

[0014] In the Neck part, a variety of feature fusion technologies and design strategies are used, including PAN (Path Aggregation Network) module, FPN (Feature Pyramid Network) module and SAM (Spatial Attention Mechanism), etc. These modules basically adopt a bottom-up and top-down approach to process and fuse feature maps from different stages and resolutions, forming a richer and more comprehensive feature representation, while maintaining efficient computing speed and low-complexity model parameters.

[0015] In the Head part, as the last layer of the network structure in the target detection algorithm, it is mainly used to predict the output of the detection box and category confidence. YOLOv5 uses CIOU_Loss as the loss function of the Bounding box. CIoU adds the loss of the detection box scale on the basis of DIoU, and adds the loss of length and width, so that the predicted box will be more consistent with the real box. Use NMS non-maximum suppression to suppress redundant boxes during detection. First, sort the confidence of all predicted boxes in descending order, select the predicted box with the highest confidence, confirm it as the correct prediction, and calculate its IOU with other predicted boxes, then remove the ones with high overlap according to IOU, and finally return the remaining predicted boxes to the first step and repeat until there are none left.

[0016] (2) Pedestrian attribute recognition model training module

[0017] The pedestrian attribute recognition model training module uses the public dataset PA-100K as training data. The PA-100K dataset is the largest dataset for pedestrian attribute recognition to date. It contains a total of 100,000 pedestrian images collected from outdoor surveillance cameras, and each image has 26 common attributes. The entire dataset is randomly divided into 80,000 training images, 10,000 validation images, and 10,000 test images. The network used in the training model is the ResNet network in the CNN model. This network is a residual network that is often used to solve image classification tasks. It solves the gradient disappearance and model degradation problems in deep networks by introducing residual connections. The core idea of ​​ResNet is to use residual blocks to build a network. Each residual block contains multiple convolutional layers, and there will be direct jump connections across adjacent layers. This jump connection allows information to be directly transferred between different layers, thereby avoiding the gradient disappearance problem.

[0018] (3) YOLOv5 model prediction module

[0019] The model prediction module is mainly used to predict the input video, which is mainly divided into five steps.

[0020] Input image: Input the image to be detected into the network. After preprocessing, the image will be scaled to a specified size (such as 640x640, 416x416, etc.) and normalized to between 0 and 1.

[0021] Feature extraction: Extract the features of the image through a convolutional neural network to obtain the target area information in the image. In YOLOv5, CSPDarknet is used as the backbone network, which consists of a series of convolutions, residual blocks, and bottleneck blocks, which can better balance the computational complexity and detection accuracy.

[0022] Final feature map processing: The feature map is divided into multiple grids, each of which is responsible for detecting an object in the image. For each grid, the algorithm outputs multiple anchor points and corresponding prediction values ​​(confidence, category probability, and bbox coordinates, etc.).

[0023] Prediction box screening: Prediction boxes are screened based on confidence and category probability, and boxes with low prediction confidence, category mismatch, or high overlap are filtered out. The non-maximum suppression (NMS) algorithm is used to remove redundant boxes.

[0024] Output result: Output the retained target box and the corresponding category to form the target detection result

[0025] (4) Pedestrian attribute recognition model prediction module

[0026] The pedestrian attribute recognition model prediction module is based on target detection. After identifying the pedestrian label, it obtains the coordinate position of the pedestrian detection box, crops the pedestrian area in the original image according to the coordinate position, pre-processes the cropped image, and loads it into the ResNet model for reasoning. The reasoning results are displayed in the video in the form of text. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The overall block diagram of monitoring security early warning technology based on computer vision

[0028] Figure 2 This is a module diagram of the application method

[0029] Figure 3 Deep learning dangerous behavior detection feature extraction model architecture based on YOLOv5

[0030] Figure 4 Specific implementation process for monitoring security early warning technology based on computer vision Specific implementation methods

[0032] In order to better describe the monitoring security early warning technology based on computer vision, the specific implementation method of this application is given below.

[0033] This project trained a safety warning recognition model based on YOLOv5 target detection in Windows system, and trained a pedestrian attribute recognition model based on ResNet network. After opening the project, select the detect.py file to enter the image path, video path or call the camera for real-time detection. The detected dangerous behavior results will be alerted and saved as pictures.

[0034] After calling the camera, the dangerous behaviors in the picture can be identified. When fighting, armed, injured, smoke, fire and other targets appear, the picture of the current picture will be output, and the saving path can be set by yourself. At the same time, the alarm music will be played, and this music can also be changed according to different targets or needs. Generally, fighting behavior is a scuffle between two or more people with large movements; armed targets include sticks, knives, and guns and have corresponding labels; injured targets are mostly people who fall to the ground; fireworks targets are mostly thick smoke and open flames. In order to realize the recognition of various dangerous behaviors, this project sets up YOLOv5 model training module, pedestrian attribute recognition model training module, YOLOv5 model prediction module, and pedestrian attribute recognition model prediction module.

[0035] (1) YOLOv5 model training

[0036] This project detects the category and location of dangerous behaviors by training deep neural networks. When training the network, it is necessary to create a corresponding training data set, so it is necessary to annotate various dangerous behavior categories (image annotation is a process of adding labels to images. Its target range can be either using only one label on the entire image or assigning multiple labels to each group of pixels in a certain image). The project uses rectangular box annotation, uses the corresponding image calibration tool, and configures the environment Python+labelimg to calibrate the target in the image. The generated TXT file follows the PASCAL VOC format (PASCAL VOC provides a complete set of standardized excellent data sets for image recognition and classification, which are used to build and evaluate algorithms for image classification, detection, and segmentation). DataSet is used to represent the folder name of the annotated data, images represents the file name of the annotated image, and labels represents the tag file format. The annotated data set is divided into 70% training set, 20% validation set, and 10% test set. Each used label is mapped to an integer value through label mapping for training, testing, and evaluation processes. The dangerous behavior image uses a pre-trained model for feature extraction, and the extracted image features are passed into our target detection model for further training. The target detection model learns the input data through data preprocessing, feature extraction, up and down sampling fusion, construction of multi-scale receptive field, classification and regression prediction, and use of NMS to filter prediction boxes. Finally, the parameters such as batch_size, epochs, optimizer, network model, and pre-trained model in the train.py file are configured to complete the training process.

[0037] (2) Pedestrian attribute recognition model training

[0038] The pedestrian attribute recognition model is trained using the public data set PA-100K. The labeled data set is divided into 80% training set, 10% validation set, and 10% test set. The ResNet network is used for model training. The batch_size, epochs, learning rate, and number of classification categories in the train.py file are configured to complete the training.

[0039] (3) YOLOv5 model prediction

[0040] After completing the training of the dangerous behavior detection model, configure the detect.py file parameters: use the best weight in the training process, that is, the best.pt file, debug the confidence threshold and IOU threshold, set the input path to the camera, and finally run the file to complete the model prediction. After connecting the camera, the input of this project is the real-time monitoring screen of the camera, and real-time detection of pedestrians, vehicles, weapons, injuries, smoke, fire, fights, drones and other targets. If weapons, fights, injuries, smoke, fire and other targets are detected, the alarm music will be played and the current screen content and prediction box will be saved.

[0041] (4) Pedestrian attribute recognition model prediction

[0042] Select the best.pt weight file with the best training effect, load the classification label pkl file, including: female, over 60 years old, 18-60 years old, under 18 years old, front, side, back, hat, glasses, handbag, shoulder bag, backpack, handheld items, short sleeves, long sleeves, top stripes, top logo, top plaid, top splicing, bottom stripes, bottom pattern, long coat, trousers, shorts, skirt & dress, boots and other 26 categories, run the detect.py file, you can predict the attributes of pedestrians.

Claims

1. Computer vision-based monitoring security early warning technology, characterized by: Detect targets such as drones, pedestrians, vehicles, weapons, injuries, smoke, fire, fighting, etc., and identify pedestrian attributes, including age, tops, bottoms, shoes, hats, glasses, bags, etc. The specific steps are as follows: Step 1) Collect and process the data set required for training the target detection model, totaling 11,000 images, including 16 types of targets; Step 2) Modify the configuration file and debug the parameters; Step 3) training the target detection model; Step 4) Use the PA-100K dataset to train a pedestrian attribute recognition model Step 5) testing the model; Step 6) Evaluate the model.

2. The computer vision-based monitoring safety early warning technology according to claim 1 is characterized in that Step 1) First, collect a dataset of 11,000 images of 16 types, including pedestrians, knife-wielding, stick-wielding, gun-wielding, fighting areas, punching, kicking, injured, motorcycles, bicycles, cars, buses, trucks, drones, smoke, and fire. Use the labelimg tool to annotate the images and divide them into training set, validation set, and test set in a ratio of 7:2:

1.

3. The computer vision-based monitoring safety early warning technology according to claim 1 is characterized in that Step 2) Use yolov5m.pt for pre-trained weights; select yolov5m.yaml for the model configuration file, nc=16; select wj.yaml for the data configuration file, nc=16; select SGD for the optimizer; and the learning rate is 0.

01.

4. The computer vision-based monitoring safety early warning technology according to claim 1 is characterized in that Step 3) Configure the parameters of the train.py file, epochs to 500, batch_size to 32, and imgsz to 640. Run train.py to start training the object detection model.

5. The computer vision-based monitoring safety early warning technology according to claim 1 is characterized in that Step 4) Use the PA-100K dataset to train the pedestrian attribute recognition model, with 300 epochs, 32 batch_size, and 224 resize. Run the train.sh file to start training the model.

6. The computer vision-based monitoring safety early warning technology according to claim 1 is characterized in that Step 5) Configure the parameters of the detect.py file. The model file uses the best training model best.pt; conf-thres is 0.5; iou-thres is 0.

6. Run detect.py to start the test.

7. The computer vision-based monitoring safety early warning technology according to claim 1 is characterized in that Step 6) Configure the parameters of the val.py file, select wj.yaml for data, use the test set of 1100 images for evaluation, use best.pt as the model file, set batch_size to 32, set conf-thres to 0.001, and set iou-thres to 0.

6. Run val.py to start the evaluation.

Images