A method for recognizing specific behaviors in a dense crowd environment

Abstract

The application discloses a specific behavior recognition method in a dense crowd environment, comprising the following steps: S1, data set acquisition, wherein the data set comprises a pedestrian detection data set and a behavior recognition data set; S2, data set preprocessing; S3, inputting images in the preprocessed pedestrian detection data set into a feature map pyramid network to extract corresponding features, generating a candidate region and class information of the candidate region through a region generation network; S4, removing overlapping targets by using a maximum value suppression algorithm; S5, performing specific behavior target recognition based on a classification recognition network of a residual network; S6, training grid parameters of the classification recognition network; S7, obtaining optimal grid parameters through step S6, and importing the optimal grid parameters into the classification recognition network, and testing through a behavior recognition data set. The method realizes the detection and recognition of specific behaviors in a dense crowd environment by fusing two stages of detection and recognition tasks for the special environment of the dense crowd.

Description

Technical Field

[0001] This invention relates to the field of identification and positioning technology, specifically to a method for identifying specific behaviors in densely populated environments. Background Technology

[0002] As artificial intelligence (AI) technology matures, target recognition technologies for pedestrians and vehicles are increasingly being applied to people's lives, such as facial recognition technology supporting face-scanning payments and license plate recognition technology supporting vehicle entry and exit registration. However, due to the diverse nature of daily life, it is often necessary to detect multiple targets simultaneously in complex crowds, and single-target recognition technologies are insufficient to solve the task of identifying specific targets in large groups. Therefore, it is essential to provide specific group behavior detection and recognition technologies for densely populated environments, which is also an inevitable stage for AI technology to improve human life. Considering the diversity of specific behaviors in complex environments, solutions need to be designed to address specific problems. This patent takes pedestrian traffic safety as its starting point and focuses on identifying the specific behavior of "smartphone addicts" on the road.

[0003] The widespread use of smartphones and their increasingly diverse functions have led to a growing dependence on them, resulting in a large number of "smartphone zombies" on the streets. This has caused frequent traffic accidents and posed a significant threat to traffic safety. Currently, there is a lack of legal constraints on the "smartphone zombie" phenomenon on the roads, relying mainly on traffic police to persuade and educate pedestrians, which is a heavy workload. Therefore, this patent proposes an automatic identification technology for "smartphone zombies" in densely populated environments to reduce the burden on traffic police, improve work efficiency, and regulate pedestrian traffic behavior. Summary of the Invention

[0004] This invention proposes a method for recognizing specific behaviors in densely populated environments, which integrates the detection and recognition tasks in two stages to achieve the detection and recognition of specific behaviors in densely populated environments.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A method for recognizing specific behaviors in densely populated environments includes the following steps:

[0007] S1. Dataset acquisition, the dataset includes pedestrian detection dataset and behavior recognition dataset;

[0008] S2, Dataset Preprocessing

[0009] S2-1. Without losing image information, standardize the dataset size, use the ImageNet training set to sample and calculate image data and then normalize it.

[0010] S2-2. Label whether pedestrians in the normalized image have a specific behavior, with 0 indicating the presence of the specific behavior and 1 indicating the absence of the specific behavior.

[0011] S3. Input the images in the preprocessed pedestrian detection dataset into the feature map pyramid network to extract the corresponding features, and then generate candidate regions and the category information of the candidate regions through the region generation network. For the generated candidate regions, use the multi-instance prediction method to predict an instance set.

[0012] S4. Use an improved maximum suppression algorithm to remove overlapping targets;

[0013] S5. Residual network-based classification and recognition network for identifying specific behavioral targets.

[0014] S6. Grid parameters for training the classification and recognition network;

[0015] S7. Obtain the optimal grid parameters through step S6 and import them into the classification and recognition network for testing using the behavior recognition dataset.

[0016] Preferably, the pedestrian detection dataset uses the CrowdHuman dataset for crowded scenes; the behavior recognition dataset is obtained by capturing images of densely populated traffic scenes.

[0017] Preferably, in step S3, the feature map pyramid network of RoI Align is used. The high-level features of the feature map pyramid network are upsampled and the low-level features are connected from top to bottom. The corresponding features are predicted for each layer.

[0018] Preferably, in step S3, a region candidate network in a pyramid structure is used to generate candidate regions and their category information. The method is as follows: first, anchor boxes are generated, each anchor box is determined to be either the foreground or the background containing the object and is classified into two categories, and bounding box regression is used to fine-tune the anchor boxes so that the selected anchor boxes are closer to the ground truth boxes.

[0019] Preferably, in step S3, for generating candidate regions, a detection function with parameter K is used to predict an instance set, as shown in the following expression:

[0020] G(b i )={g j ∈G|IOU(b i ,g j )≥θ} (1)

[0021]

[0022] Where b iG(b) represents the i-th candidate region proposal box. i ) indicates the relationship with b i The corresponding set of real instances is the ground truth set, G represents the set of all ground truth boxes, and K represents the maximum number of instances in the candidate box, denoted as G(b i The largest cardinality of P(b) i ) is a set of predicted instances, c i It is a confidence label, l i θ represents the corresponding position, and θ is a threshold for the given joint intersection ratio.

[0023] Preferably, the specific method of step S4 is as follows: select the candidate region with the highest confidence from the multiple candidate regions obtained in step S3 as the first bounding box, and then select one from the remaining candidate regions as the second bounding box. If the two bounding boxes come from the same candidate region, skip the suppression step; otherwise, calculate the joint intersection ratio of the two bounding boxes using the maximum suppression algorithm. If the value is greater than the threshold, the second bounding box is removed. Then repeat the above operation on the remaining candidate regions in turn until all candidate regions have been traversed, and determine the final candidate regions.

[0024] Preferably, the classification and recognition network includes a first-stage detection network and a second-stage classification network. The first-stage detection network uses a three-layer fully connected convolutional neural network for preliminary classification, directly classifying targets with a probability greater than 0.1 into the "playing with mobile phones" category, and setting targets with a probability less than 0.1 as pending. The second-stage classification network uses a three-layer ResNet-50 framework network with two neurons to further classify the pending targets.

[0025] Preferably, in step S6, the θ of the first-stage detection network... d The parameters are trained using the EMD (Earth Map Distance) function to minimize the prediction set and P(b). i (θ d )) and b i The corresponding ground truth instance set G(b) i The distance between them is expressed as follows:

[0026]

[0027]

[0028]

[0029]

[0030] Where π represents a specific permutation (1, 2, ..., K), and the k-th term is π. k , It is the k-th term π k Real-world examples of ground truth boxes, L cls (·) and L reg (·) represent the classification loss and the bounding box regression loss, respectively, c i It is a confidence label, l i R represents the corresponding position, and R is the Smooth L1 loss function.

[0031] Preferably, in step S6, the second stage identifies the θ of the network. c and θ R Parameter training,

[0032] θ is calculated using the cross-entropy loss function. c The parameters are expressed as follows:

[0033]

[0034] Where y is the network's output vector;

[0035]

[0036] Where x is the network's output vector and class is the real label.

[0037] This invention has the following characteristics and beneficial effects:

[0038] Using the above technical solution, in the pedestrian detection stage, a Feature Map Pyramid Network (FPN), a Region Generation Network (RPN), and a multi-instance prediction method are used for prediction. For the obtained results, an improved Set-NMS algorithm is used to remove overlapping boxes. In the target recognition stage, a three-layer fully connected convolutional neural network is used for initial screening to determine if a person is using a mobile phone. Targets for which no mobile phone was detected are then input into a ResNet-50 framework network for further subdivision. Experiments on a self-made dataset demonstrate that this method can accurately identify specific behaviors and outperforms traditional detection methods in terms of accuracy. Therefore, it can be widely applied in practical applications, such as helping to reduce the workload of traffic police, improve work efficiency, and regulate pedestrian traffic behavior. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating an embodiment of a specific behavior recognition method in a densely populated environment according to the present invention.

[0041] Figure 2 This is a flowchart of the classification and recognition model in an embodiment of the present invention.

[0042] Figure 3 This is a flowchart of the Set-NMS algorithm in an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 , 2 Section 3 provides a further detailed description of the specific crowd behavior identification method (pedestrian detection method and behavior recognition method) provided by the present invention in dense crowd environments.

[0045] A method for recognizing specific behaviors in densely populated environments includes the following steps:

[0046] Specifically, in this embodiment, the specific behavior in the applied scenario refers to the behavior of looking down at a mobile phone in a crowd.

[0047] S1. Dataset Acquisition: In the pedestrian detection stage, the CrowdHuman dataset, designed for crowded scenarios, was used to acquire the pedestrian detection dataset. In the behavior recognition stage, the behavior recognition dataset was acquired using specific locations such as the student activity center, Fred Square, and Wenze Square in the teaching area of ​​Hangzhou Dianzi University. An iPhone 10 was used to capture images of densely populated traffic scenes during two time periods: 9:00-11:00 AM and 8:00-10:00 PM. The images were taken following three principles: being in a densely populated environment, avoiding camera shake, and ensuring that the upper body of the pedestrian is visible in the image. A total of 419 images were captured for dataset creation, and 1418 pedestrians were then extracted from these images for the subsequent behavior recognition stage.

[0048] S2. Dataset preprocessing:

[0049] S2-1. Without losing image information, standardize the size of the above image dataset and use the data obtained by sampling from the ImageNet training set: mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225], where mean represents the mean and std represents the variance; normalize the dataset.

[0050] S2-2. Label whether pedestrians in the normalized image are looking at their phones: 0 for looking at their phones and 1 for not looking at their phones.

[0051] S3. Pedestrian detection in complex scenes using multi-instance prediction: The preprocessed image is input into the Feature Map Pyramid Network (FPN) of RoI Align to extract the corresponding features fpn_fms. Then, the Region Generation Network (RPN) generates candidate regions rpn_rois and the category information of the candidate regions. The multi-instance prediction method is used to predict an instance set for the generated candidate regions.

[0052] G(b i )={g j ∈G|IOU(b i ,g j )≥θ} (1)

[0053]

[0054] Where b i G(b) represents the i-th candidate proposal box. i ) indicates the relationship with b i The corresponding set of real instances is the ground truth set, G represents the set of all ground truth boxes, and K represents the maximum number of instances in the candidate box, denoted as G(b i The largest cardinality of P(b) i ) is a set of predicted instances, c i It is a confidence label, l i This corresponds to the position. θ is a threshold for the given joint intersection (IOU) ratio, which is 0.01 in this embodiment.

[0055] Specifically, the Feature Map Pyramid Network (FPN) upsamples high-level features and connects low-level features from top to bottom, and each layer makes predictions; the Region Generation Network (RPN) is essentially a classless obejct detector based on a sliding window, and the Joint Intersection (IOU) ratio refers to the ratio of the overlapping regions of two detection boxes to the union of the two regions.

[0056] S4. Target Overlap Resolution: An improved Set-NMS (Set Maximum Suppression) algorithm is used to remove overlapping targets. The specific steps are as follows: From the multiple candidate proposal boxes generated in the above process, select the first bounding box with the highest confidence. Then, select a second bounding box from the remaining candidate regions. If the two boxes come from the same candidate proposal box, skip the suppression step; otherwise, calculate the Intersection over Union (IOU) ratio of the two boxes. If the value is greater than a threshold, the bounding box is removed. Repeat the above operation for the remaining candidate regions until all candidate regions have been traversed, determining the final candidate proposal boxes. At this point, the pedestrian detection phase is complete.

[0057] S5. Residual network-based classification and recognition network for identifying specific behavioral targets.

[0058] This method uses a multi-instance prediction approach to identify specific targets, such as whether someone is using a mobile phone. A fast but less accurate three-layer fully connected convolutional neural network is used for initial classification. Targets with a probability greater than 0.1 are directly classified into the "using a mobile phone" category, while targets with a probability less than 0.1 are left undetermined. For undetermined targets, a slower but more accurate classification network is used for further classification. (The classification network is a ResNet-50 framework network with fully connected layers modified to two neurons.)

[0059] S6. Grid parameters for training the classification and recognition network;

[0060] The pedestrian detection network is trained according to formula (3) with θ. d parameter:

[0061] Minimize the prediction set and P(b) using the EMD function (Earth Map Distance). i (θ d )) and b i The corresponding ground truth instance set G(b) i Distance between:

[0062]

[0063]

[0064]

[0065]

[0066] Where π represents a specific permutation (1, 2, ..., K), and the k-th term is π. k , It is the k-th term πk Real-world examples of ground truth boxes, L cls (·) and L reg (·) represent the classification loss and the bounding box regression loss, respectively, c i It is a confidence label, l i R represents the corresponding position, and R is the Smooth L1 loss function.

[0067] The θ of the classification and recognition network is trained according to formula (7). c θ R parameter:

[0068] θ is calculated using the following cross-entropy loss function. c :

[0069]

[0070] Where y is the network's output vector;

[0071] θ is calculated using the following cross-entropy loss function. R :

[0072]

[0073] Where x is the network's output vector and class is the real label.

[0074] S7. Testing new samples with optimal parameters: Based on step S6, the pedestrian detection network parameters θ are obtained by training using formula (3). d The behavior recognition parameters θ are obtained by training using formulas (7) and (8). c θ R Then, using this set of parameters, steps S3, S4, and S5 are executed to obtain the detection results.

[0075] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for recognizing specific behaviors in densely populated environments, characterized in that, Includes the following steps: S1. Dataset acquisition, the dataset includes pedestrian detection dataset and behavior recognition dataset; S2, Dataset Preprocessing S2-1. Without losing image information, standardize the dataset size, use the ImageNet training set to sample and calculate image data and then normalize it. S2-2. Label whether pedestrians in the normalized image have a specific behavior, with 0 indicating the presence of the specific behavior and 1 indicating the absence of the specific behavior. S3. Input the images in the preprocessed pedestrian detection dataset into the feature map pyramid network to extract the corresponding features, and then generate candidate regions and their category information through the region generation network. In step S3, the Region Generation Network (RPN) is used to generate candidate regions and their category information. The method is as follows: First, anchor boxes are generated. Each anchor box is determined to be either foreground or background containing an object and is classified binary. Bounding box regression is used to fine-tune the anchor boxes so that the selected anchor boxes are closer to the ground truth boxes. For the generated candidate regions, a detection function with parameter K is used to predict an instance set, as shown in the following expression: (1) (2) in Indicates the first The candidate region proposal box, Indicates and The corresponding set of real instances, the ground truth set. This represents the set of all ground truth boxes. This represents the maximum number of instances in the candidate box, and also indicates... The largest cardinality, It is a set of predicted instances. It is a confidence label. It is the corresponding position. It is a threshold for the given joint intersection ratio; S4. Use an improved maximum suppression algorithm to remove overlapping targets; The method for removing overlapping targets is as follows: Select the candidate region with the highest confidence from the multiple candidate regions obtained in step S3 as the first bounding box, and then select one from the remaining candidate regions as the second bounding box. If the two bounding boxes come from the same candidate region, skip the suppression step; otherwise, calculate the joint intersection ratio of the two bounding boxes using the maximum suppression algorithm. If the value is greater than the threshold, the second bounding box is removed. Then repeat the above operation on the remaining candidate regions in turn until all candidate regions have been traversed to determine the final candidate regions. S5. A classification and recognition network based on residual networks is used to identify specific behavioral targets. The classification and recognition network includes a first-stage detection network and a second-stage classification network. The first-stage detection network uses a three-layer fully connected convolutional neural network for preliminary classification. Targets with a probability greater than 0.1 are directly classified into the "playing on a mobile phone" category, while targets with a probability less than 0.1 are set as pending. The second-stage classification network uses a three-layer ResNet-50 framework network with two neurons to further classify the pending targets. S6. Grid parameters for training the classification and recognition network; S7. Obtain the optimal grid parameters through step S6 and import them into the classification and recognition network for testing using the behavior recognition dataset.

2. The method for recognizing specific behaviors in densely populated environments according to claim 1, characterized in that, The pedestrian detection dataset uses the CrowdHuman dataset for crowded scenes; the behavior recognition dataset is obtained by capturing images of densely populated traffic scenes.

3. The method for recognizing specific behaviors in densely populated environments according to claim 1, characterized in that, In step S3, the feature map pyramid network of RoI Align is used. The high-level features of the feature map pyramid network are upsampled and the low-level features are connected from top to bottom. The corresponding features are predicted at each layer.

4. The method for recognizing specific behaviors in densely populated environments according to claim 1, characterized in that, In step S6, the first stage detection network... Parameters are trained using the graph distance function. Minimize the prediction set and Corresponding ground truth instance set The distance between them is expressed as follows: (3) (4) (5) (6) in, Represents a specific permutation , No. Item for , It is the first item A real-world example: the ground truth box. and Let these represent the classification loss and the bounding box regression loss, respectively. It is a confidence label. It is the corresponding position. It is the Smooth L1 loss function.

5. The method for recognizing specific behaviors in densely populated environments according to claim 4, characterized in that, In step S6, the classification and recognition network and Parameter training, Calculate using the cross-entropy loss function The parameters are expressed as follows: (7) in, It is the network's output vector; (8) in, It is the network's output vector. It's a real label.

Images