A Multi-Label Pedestrian Abnormal Behavior Recognition Method Based on Computer Vision

Abstract

This invention discloses a multi-label pedestrian abnormal behavior recognition method based on computer vision, belonging to the field of computer vision application technology. This method acquires pedestrian behavior sample video data from surveillance videos; preprocesses the acquired pedestrian behavior sample video data to obtain a final dataset; divides the final dataset proportionally into training and test sets to build a BDNet dual-branch convolutional neural network classification model to classify and recognize pedestrian abnormal behaviors; and uses the trained BDNet dual-branch convolutional neural network classification model to identify pedestrian actions. This method uses computer vision algorithms to automatically extract overall human features and facial detail features, fits multi-attribute pedestrian action types through multiple linear regression, and accurately predicts various pedestrian violations and provides real-time warnings in a non-contact, real-time manner, playing an important role in pedestrian safety detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision application technology; and more particularly to a multi-label pedestrian abnormal behavior recognition method based on computer vision. Background Technology

[0002] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and is not necessarily to be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

[0003] In the field of public safety, intelligent video surveillance has received increasing attention in recent years. Pedestrian behavior detection plays a crucial role in detecting various violations, such as pedestrians running, fighting, or punching. Although abnormal behavior detection has been successfully applied in some areas, it still faces a series of challenges. The earliest research on human motion recognition dates back to the 1970s, when psychologists conducted experiments where people in dark environments had light spots attached to their joints, capturing the movement of these bright spots on the body to obtain motion information. Many researchers have conducted extensive and detailed studies in this field. For example, some researchers have analyzed pedestrian behavior and built global models to detect abnormal events, such as social force models and interactive energy potentials, but these methods rely on complex manual functions and have significant limitations. Other research focuses on extracting and analyzing carefully selected descriptors of targets, such as trajectories, histogram optical flow, directional gradient histograms, and spatiotemporal directional energy. In a specific use case, trajectory analysis is effective as a traffic monitoring or illegal exclusion zone. However, this approach requires the use of accurate upstream pedestrian detection and tracking algorithms, which can become overly complex and unreliable in situations with a large number of people. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a multi-label pedestrian abnormal behavior recognition method based on computer vision. It obtains pedestrian behavior sample data from surveillance videos frame by frame, automatically extracts the overall movement features and head detail features of pedestrians through computer vision algorithms, and enhances the robustness and real-time performance of the algorithm by utilizing multi-scale information and a lightweight deep learning network. This significantly reduces the computational load of the network model and lowers the hardware requirements and costs of the algorithm.

[0005] To solve the aforementioned technical problem, the present invention adopts the following technical solution: a multi-label pedestrian abnormal behavior recognition method based on computer vision, comprising the following steps:

[0006] S01) Obtain pedestrian behavior sample video data;

[0007] S02) Preprocess the pedestrian behavior sample video data obtained in step S01) to obtain the final dataset; this step specifically includes:

[0008] S21) Clean the pedestrian behavior sample video data collected in step S01) and remove unqualified video segments with no pedestrians or unclear images to obtain cleaned pedestrian behavior video sample data.

[0009] S22) The pedestrian behavior sample video data obtained in step S21) is segmented into single, unobstructed pedestrian behavior image data, and each sample is labeled. The specific process includes the following steps:

[0010] S221) Video frame segmentation: The pedestrian behavior sample video data obtained in step S21) is divided into image data at equal intervals, and each image contains pedestrian behavior information.

[0011] S222) Extract a single human image from the image data obtained in step S221 according to the pre-marked bounding box. The single human image data contains pedestrian behavior information.

[0012] S223) Label the single portrait data obtained in step S222). The labels are divided into two categories: one category represents the overall state of the portrait, which belongs to the single label classification task; the other category represents the detailed state of the portrait, which belongs to the multi-label classification task.

[0013] S224) Name folders according to scene, store single portrait image data obtained in step S222), and write their corresponding labels into a CSV label file. The CSV file contains image path, scene information label, overall portrait status information and portrait detail status information.

[0014] The single portrait data obtained in step S225) and step S221) and the CSV label file obtained in step S224) are used as the final dataset;

[0015] S03) Divide the final dataset obtained in step S02) into training set samples and test set samples according to the proportion;

[0016] S04) Build a BDNet dual-branch convolutional neural network classification model to classify and identify abnormal pedestrian behaviors;

[0017] The BDNet dual-branch convolutional neural network classification model includes a feature extraction part, a global state classifier, and a detail state classifier. The feature extraction part uses a pre-trained ResNet18 model, with layer 5 being the output of this module. The feature shape is uniformly set to N*1000 before being used as the output of this part. The global state classifier contains two linear convolutions, which sample the features into N*256 and N*5 sizes respectively, with the N*5 size feature being one of the total network outputs. The detail state classifier also contains two linear convolutions, which sample the features into N*256 and N*4 sizes respectively, with the N*4 size feature being one of the total network outputs.

[0018] S05) Use the trained BDNet dual-branch convolutional neural network classification model to identify pedestrian actions.

[0019] Furthermore, the BDNet dual-branch convolutional neural network classification model uses the Softmax activation function for the N*5 global features and the Sigmoid activation function for the N*4 detail features.

[0020] Furthermore, the initial learning rate of the BDNet dual-branch convolutional neural network classification model is 0.0001, and the learning rate is updated based on the validation results using the callback function ReduceLROnPlateau in Keras.

[0021] Furthermore, the loss function of the BDNet dual-branch convolutional neural network classification model is the binary cross-entropy loss function.

[0022] Further, step S05) specifically involves: setting the number of iterations to Z, training the BDNet dual-branch convolutional neural network multi-label classification model using training set samples, using the trained BDNet model to predict the human action test set samples to be classified, and then verifying the prediction against the actual labels to calculate the overall prediction accuracy.

[0023] Further, step S01) specifically involves: installing cameras in the specified scene and ensuring that the field of view is complete and clear; arranging multiple pedestrians to pass through the above-mentioned locations and make different abnormal actions; filming the passing pedestrians to obtain pedestrian behavior sample video data.

[0024] Further, in step S03), the data is subjected to 5-class classification training and verification, and the classification is verified based on the scene information labels stored in the CSV label file obtained in step S224.

[0025] The beneficial effects of this invention are as follows: 1. This invention utilizes bounding boxes in images to crop multi-person portrait images in complex scenes, obtaining a single, unobstructed single-person portrait image, eliminating the interference of complex backgrounds on the target subject, and ensuring the accuracy of subsequent multi-label classification and the anti-interference ability of the model.

[0026] 2. This invention uses cropped single pedestrian images as a dataset to construct a multi-label, multi-scale, dual-branch classification neural network. The network automatically extracts overall human body features and facial detail features, and fits the multi-attribute action types of pedestrians through multiple linear regression. This method achieves multi-label classification of pedestrian behavior using surveillance video, providing a new approach for pedestrian action recognition and detection. Attached Figure Description

[0027] Figure 1 A flowchart of a multi-label pedestrian abnormal behavior recognition method based on computer vision provided in an embodiment of the present invention;

[0028] Figure 2 This is a pedestrian behavior annotation map provided in an embodiment of the present invention;

[0029] Figure 3 This is a pedestrian behavior detail annotation map provided in an embodiment of the present invention;

[0030] Figure 4 A multi-label pedestrian behavior classification framework diagram provided in an embodiment of the present invention;

[0031] Figure 5 This is a diagram of the architecture of a multi-label dual-branch convolutional neural network model provided in an embodiment of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0033] Example 1

[0034] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0036] This embodiment discloses a multi-label pedestrian abnormal behavior recognition method based on computer vision, such as... Figure 1 As shown, it includes the following steps:

[0037] Step 1 involves obtaining sample video data of pedestrian behavior from surveillance videos, including the following steps:

[0038] Step 1.1: Install cameras in five common locations, including corridors, hallways, lobbies, equipment rooms, and restroom entrances, ensuring that the field of view is complete and clear. Arrange for multiple employees to pass through these locations and perform different unusual actions to record videos of the passing pedestrians and obtain pedestrian behavior sample video data.

[0039] In this embodiment, all the sample data collected are from our company's scenarios, and all the personnel involved are our company's employees.

[0040] Step 2: The specific process of preprocessing the pedestrian behavior sample video data obtained in Step 1 to obtain the final dataset includes the following steps:

[0041] Step 2.1: Clean the pedestrian behavior video sample data collected in Step 1.1, remove unqualified video segments with no pedestrians or unclear images, and obtain cleaned pedestrian behavior video sample data.

[0042] Step 2.2: Segment the pedestrian behavior sample video data obtained in Step 2.1 into single, unobstructed pedestrian behavior image data, and label each sample. The specific process includes the following steps:

[0043] Step 2.2.1: Video Frame Segmentation: The pedestrian behavior sample video data obtained in Step 2.1 is segmented into image data at equal intervals, and each image contains pedestrian behavior information;

[0044] Step 2.2.2: Extract a single human figure with a small amount of background from the image data obtained in step 2.2.1 according to the pre-marked bounding box. This single human figure data contains pedestrian behavior information.

[0045] Step 2.2.3: Label the single-image data obtained in Step 2.2.2. Labels are divided into two categories: one category represents the overall state of the image (Break / Fight / Run / Tumble / Walk), which is a single-label classification task, such as... Figure 2 As shown, this is a pedestrian behavior annotation map provided in an embodiment of the present invention; another type is representing the detailed state of a person's face (Head / Helmet / Calling / Smoking), which belongs to a multi-label classification task, such as... Figure 3 The image shown is a pedestrian behavior detail annotation diagram provided in an embodiment of the present invention;

[0046] Step 2.2.4: As described in Step 1.1, this dataset contains 5 scenes, and folders are named according to scene to store the single portrait image data obtained in Step 2.2.2, and their corresponding labels are written to a CSV file. This CSV file contains image paths, scene information labels, overall portrait status information, and portrait detail status information;

[0047] Step 2.2.5: The single portrait data obtained in Step 2.2.1 and the CSV label file obtained in Step 2.2.4 are used as the final dataset.

[0048] Step 3: The specific process of dividing the final dataset obtained in Step 2 into training set samples and test set samples according to the proportions includes the following steps:

[0049] The data was trained and validated using five classification methods. The classification was performed based on the scene information labels stored in the CSV label file obtained in step 2.2.4.

[0050] Step 4: The specific process of building a BDNet (Behavior Detection Network) dual-branch convolutional neural network classification model to classify and identify abnormal pedestrian behavior includes the following steps:

[0051] like Figure 5 As shown, the convolutional neural network parameters are set as follows: the network input is a 256*128 RGB image. The overall network framework consists of three parts: a feature extraction part, which uses a pre-trained ResNet18 model, where layer 5 is the output of this module, and the feature shape is uniformly N*1000 (where N is the batch size during training, the same below) before being used as the output of this part; a global state classifier, which contains two linear convolutions, sampling the features to N*256 and N*5 respectively, with the N*5 size feature being one of the total network outputs; and a detail state classifier, which also contains two linear convolutions, sampling the features to N*256 and N*4 respectively, with the N*4 size feature being one of the total network outputs. Activation functions: Softmax activation function is used for N*5 global features, and Sigmoid activation function is used for N*4 detail features. Learning rate: The initial learning rate is 0.0001, and the learning rate is updated based on the validation results using the Keras callback function ReduceLROnPlateau. Loss function: Binary cross-entropy loss function (BCEWithLogitsLoss) is used for all loss functions.

[0052] Step 5: The specific process of using the trained BDNet dual-branch convolutional neural network classification model to identify pedestrian actions is as follows:

[0053] The number of iterations is set to Z. The BDNet dual-branch convolutional neural network multi-label classification model is trained using training set samples. The trained BDNet model is then used to predict the human action test set samples to be classified. After the prediction is completed, it is checked against the actual labels, and the overall prediction accuracy is calculated.

[0054] like Figure 4 The diagram shown is a multi-label pedestrian behavior classification framework provided in this embodiment. Safety helmet, smoking, making a phone call, and face (prior) are detailed states, while smashing a wall, running, walking, fighting, and falling are overall states. The multi-label pedestrian abnormal behavior recognition method described in this embodiment integrates detailed states and overall states.

[0055] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-label pedestrian abnormal behavior recognition method based on computer vision, characterized in that: Includes the following steps: S01) Obtain pedestrian behavior sample video data; S02) Preprocess the pedestrian behavior sample video data obtained in step S01) to obtain the final dataset; this step specifically includes: S21) Clean the pedestrian behavior sample video data collected in step S01) and remove unqualified video segments with no pedestrians or unclear images to obtain cleaned pedestrian behavior video sample data. S22) The pedestrian behavior sample video data obtained in step S21) is segmented into single, unobstructed pedestrian behavior image data, and each sample is labeled. The specific process includes the following steps: S221) Video frame segmentation: The pedestrian behavior sample video data obtained in step S21) is divided into image data at equal intervals, and each image contains pedestrian behavior information. S222) Extract a single human image from the image data obtained in step S221 according to the pre-marked bounding box. The single human image data contains pedestrian behavior information. S223) Label the single portrait data obtained in step S222). The labels are divided into two categories: one category represents the overall state of the portrait, which belongs to the single label classification task; the other category represents the detailed state of the portrait, which belongs to the multi-label classification task. S224) Name folders according to scene, store single portrait image data obtained in step S222), and write their corresponding labels into a CSV label file. The CSV file contains image path, scene information label, overall portrait status information and portrait detail status information. The single portrait data obtained in step S225) and step S221) and the CSV label file obtained in step S224) are used as the final dataset; S03) Divide the final dataset obtained in step S02) into training set samples and test set samples according to the proportion; S04) Build a BDNet dual-branch convolutional neural network classification model to classify and identify abnormal pedestrian behaviors; The BDNet dual-branch convolutional neural network classification model includes a feature extraction part, a global state classifier, and a detail state classifier. The feature extraction part uses a pre-trained ResNet18 model, with layer 5 being the output of this module. The feature shape is uniformly set to N*1000 before being used as the output of this part. The global state classifier contains two linear convolutions, which sample the features into N*256 and N*5 sizes respectively, with the N*5 size feature being one of the total network outputs. The detail state classifier also contains two linear convolutions, which sample the features into N*256 and N*4 sizes respectively, with the N*4 size feature being one of the total network outputs. S05) Use the trained BDNet dual-branch convolutional neural network classification model to identify pedestrian actions.

2. The multi-label pedestrian abnormal behavior recognition method based on computer vision according to claim 1, characterized in that: The BDNet dual-branch convolutional neural network classification model uses the Softmax activation function for N*5 global features and the Sigmoid activation function for N*4 detail features.

3. The multi-label pedestrian abnormal behavior recognition method based on computer vision according to claim 1, characterized in that: The initial learning rate of the BDNet dual-branch convolutional neural network classification model is 0.0001. The learning rate is updated based on the validation results using the callback function ReduceLROnPlateau in Keras.

4. The multi-label pedestrian abnormal behavior recognition method based on computer vision according to claim 1, characterized in that: The loss function used in the BDNet dual-branch convolutional neural network classification model is the binary cross-entropy loss function.

5. The multi-label pedestrian abnormal behavior recognition method based on computer vision according to claim 1, characterized in that: Step S05) Specifically: Set the number of iterations to Z, train the BDNet dual-branch convolutional neural network multi-label classification model using training set samples, use the trained BDNet model to predict the human action test set samples to be classified, and after the prediction is completed, check it against the actual label and calculate the overall prediction accuracy.

6. The multi-label pedestrian abnormal behavior recognition method based on computer vision according to claim 1, characterized in that: Step S01) Specifically involves: installing cameras in the specified scene and ensuring that the field of view is complete and clear; arranging multiple pedestrians to pass through the above-mentioned locations and make different abnormal actions; filming the passing pedestrians; and obtaining pedestrian behavior sample video data.

7. The multi-label pedestrian abnormal behavior recognition method based on computer vision according to claim 1, characterized in that: Step S03) Perform 5-class classification training and verification on the data. The classification is based on the scene information labels stored in the CSV label file obtained in step S224) for classification verification.

Images