An armed person identification method and system fusing equipment and behavior features

Abstract

A kind of armed personnel identification method and system fusing equipment and behavior characteristics, method includes the following steps: obtaining equipment detection dataset, equipment detection model is trained based on the equipment detection dataset;Armed personnel behavior video stream data is obtained, and armed personnel behavior identification training sample set is constructed based on the video stream data;Armed personnel behavior identification model is trained based on the armed personnel behavior identification training sample set;Each frame image input trained equipment detection model in the video stream to be identified is extracted, and the equipment detection result of each frame image is obtained;The skeletal joint data of each frame image in the video stream to be identified is extracted;The skeletal joint data is input into trained armed personnel behavior identification model, and the personnel behavior identification result of each frame image in the video stream is obtained;Based on the equipment detection result and behavior identification result, the dangerous coefficient of each frame image personnel is calculated, and if dangerous coefficient is higher than preset threshold, then the personnel is judged as armed personnel.

Description

Technical Field

[0001] The present invention relates to the technical field of armed personnel identification, and in particular to an armed personnel identification method and system integrating equipment and behavior characteristics. Background Art

[0002] In actual counter-terrorism reconnaissance, there's no clear definition of armed personnel, making it difficult to accurately identify them solely through target detection. Armed personnel have distinct characteristics not only in their physical appearance and equipment, but also in their many behavioral traits, such as standing or crouching. Existing recognition methods often only recognize targets from a single direction, resulting in low accuracy and difficulty in accurately identifying armed personnel. Summary of the Invention

[0003] In view of the above analysis, the embodiments of the present invention aim to provide a method and system for identifying armed personnel that integrates equipment and behavioral characteristics, so as to solve the problem that the existing recognition accuracy is low and it is difficult to accurately identify armed personnel.

[0004] In one aspect, an embodiment of the present invention provides a method for identifying armed personnel by integrating equipment and behavioral characteristics, comprising the following steps:

[0005] Acquire an equipment detection dataset, and train an equipment detection model based on the equipment detection dataset;

[0006] Obtaining armed personnel behavior video stream data, constructing an armed personnel behavior recognition training sample set based on the video stream data; and training an armed personnel behavior recognition model based on the armed personnel behavior recognition training sample set;

[0007] Extract each frame of the video stream to be identified and input it into a trained equipment detection model to obtain an equipment detection result for each frame of the image; extract the skeletal joint point data of each frame of the video stream to be identified; input the skeletal joint point data into a trained armed personnel behavior recognition model to obtain a personnel behavior recognition result for each frame of the video stream; based on the equipment detection result and the behavior recognition result, calculate the risk factor of the personnel in each frame of the image; if the risk factor is higher than a preset threshold, the personnel is judged to be an armed personnel.

[0008] Based on the further improvement of the above method, the risk factor of the person in each frame image is calculated as follows:

[0009] Danger=P i IoU i +P act , where P i Indicates the confidence of the detected i-th equipment, IoU i represents the intersection-over-union ratio of the detected i-th equipment and human ROI, P actIndicates the dangerousness of the behavior recognition result.

[0010] Furthermore, the risk level of the behavior recognition result is calculated according to the following formula:

[0011] Among them, P j represents the confidence of the j-th behavior category, β j represents the risk factor of the jth behavior category, Indicates the number of behavior categories.

[0012] Furthermore, the equipment detection model is a dynamic neural network model; the dynamic neural network model includes a first subnetwork and a second subnetwork, the first subnetwork is used to detect humans in the image; when the first subnetwork detects that the image contains humans, the human ROI is extracted and transmitted to the second subnetwork; the second subnetwork is used to perform equipment detection using classifiers at different network depths according to different image resolutions; the dynamic neural network model is trained based on the equipment detection data set to obtain a trained armed personnel equipment detection model.

[0013] Furthermore, constructing a training sample set for armed personnel behavior recognition based on the video stream data includes:

[0014] Extracting skeletal joint point data of each frame of the video stream data; adding a behavior label to each frame of the image; and matching the behavior label with the skeletal joint point data to obtain an initial training sample set;

[0015] The behavior labels in the initial training sample set are smoothed to obtain a behavior recognition training sample set.

[0016] Furthermore, label smoothing is performed on the behavior labels in the initial training sample set to obtain a behavior recognition training sample set, including:

[0017] Perform overall smoothing on all behavior labels in the initial training sample set;

[0018] Determine the behavior transition frame in the initial training sample set, and perform intra-group behavior label smoothing on a group of images before the behavior transition frame;

[0019] The skeleton joint point data includes the confidence of the skeleton joint points; the behavior label of each image is confidence smoothed based on the confidence of the skeleton joint points to obtain a behavior recognition training sample set.

[0020] Furthermore, the following formula is used to perform overall smoothing on all behavior labels in the initial training sample set:

[0021]

[0022] Among them, Label represents the sample behavior label, represents the number of categories, and ε represents the smoothing parameter.

[0023] Furthermore, a group of images before the behavior conversion frame are smoothed with behavior labels within the group, including:

[0024] For each behavior conversion frame, determine the active index and target index of the image group before conversion according to the label value of the image frame before the behavior conversion frame and the label value of the behavior conversion frame;

[0025] According to the formula Labels[j][active index] = label maximum value * (ij) / k, calculate the label value corresponding to the active index in the labels of the k images before the behavior conversion frame;

[0026]

[0027] Calculate the label value corresponding to the target index in the labels of the k images before the behavior conversion frame;

[0028] Where j = ik, i-(k-1),...i-1, the i-th frame is the behavior transition frame, Labels[j][active index] represents the label value corresponding to the active index in the behavior label of the j-th frame image; Labels[j][target index] represents the label value corresponding to the target index in the behavior label of the j-th frame image, the active index is the index of the maximum value of the label in the behavior label of the i-1-th frame image, and the target index is the index of the maximum value of the label in the behavior label of the i-th frame image.

[0029] Furthermore, the confidence smoothing of the behavior label of each image is performed based on the confidence of the skeleton joint points, including:

[0030] For each image frame, the confidence of the main skeletal joints is set to 1, the confidence of other skeletal joints remains unchanged, and the mean confidence of all skeletal joints is calculated;

[0031] The mean of the confidence levels is multiplied by the label value of the frame image to obtain a smoothed label of the frame image based on the confidence levels.

[0032] On the other hand, an embodiment of the present invention provides an armed personnel identification system that integrates equipment and behavioral characteristics, including the following modules:

[0033] An equipment detection model training module is used to obtain an equipment detection data set and train an equipment detection model based on the equipment detection data set;

[0034] A behavior recognition model training module is used to obtain video stream data of armed personnel behavior, construct an armed personnel behavior recognition training sample set based on the video stream data, and train an armed personnel behavior recognition model based on the armed personnel behavior recognition training sample set;

[0035] The armed personnel identification module is used to extract each frame of the video stream to be identified and input it into a trained equipment detection model to obtain an equipment detection result for each frame of the image; extract the skeletal joint point data of each frame of the video stream to be identified; input the said skeletal joint point data into a trained armed personnel behavior recognition model to obtain a personnel behavior recognition result for each frame of the video stream; based on the said equipment detection result and behavior recognition result, calculate the risk factor of the person in each frame of the image; if the risk factor is higher than a preset threshold, the person is judged to be an armed person.

[0036] Compared with the existing technology, the present invention combines equipment characteristics and behavioral characteristics by constructing and training equipment detection models and armed personnel behavior recognition models, thereby achieving efficient and accurate identification of armed personnel, providing technical support for accurate and efficient counter-terrorism.

[0037] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0039] Figure 1 This is a flow chart of a method for identifying armed personnel by integrating equipment and behavioral characteristics according to an embodiment of the present invention;

[0040] Figure 2 A block diagram of an armed personnel identification system integrating equipment and behavioral characteristics for implementing the present invention;

[0041] Figure 3 This is a classification diagram of armed personnel behavior according to an embodiment of the present invention;

[0042] Figure 4 Skeletal joint points identified by the posture recognition algorithm of an embodiment of the present invention;

[0043] Figure 5 Part of the label data before intra-group label smoothing in an embodiment of the present invention;

[0044] Figure 6This is part of the label data after smoothing of the intra-group labels in an embodiment of the present invention;

[0045] Figure 7 This is part of the label data after confidence smoothing according to an embodiment of the present invention;

[0046] Figure 8 This is a spatiotemporal graph of skeletal joints according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0048] One embodiment of the present invention discloses a method for identifying armed personnel by integrating equipment and behavior characteristics, such as Figure 1 As shown, the following steps are included:

[0049] S1. Obtain an equipment detection dataset, and train an equipment detection model based on the equipment detection dataset;

[0050] S2. Obtaining video stream data of armed personnel behavior, constructing a training sample set for armed personnel behavior recognition based on the video stream data; and training an armed personnel behavior recognition model based on the training sample set for armed personnel behavior recognition;

[0051] S3. Extract each frame of the video stream to be identified and input it into a trained equipment detection model to obtain an equipment detection result for each frame of the image; extract the skeletal joint point data of each frame of the video stream to be identified; input the skeletal joint point data into a trained armed personnel behavior recognition model to obtain a personnel behavior recognition result for each frame of the video stream; based on the equipment detection result and the behavior recognition result, calculate the risk factor of the personnel in each frame of the image; if the risk factor is higher than a preset threshold, the personnel is judged to be an armed personnel.

[0052] The present invention combines equipment characteristics and behavioral characteristics by constructing and training an equipment detection model and an armed personnel behavior recognition model, thereby achieving efficient and accurate identification of armed personnel, providing technical support for accurate and efficient counter-terrorism.

[0053] In actual counter-terrorism reconnaissance work, the equipment carried by armed personnel is small and difficult to identify at a long distance. In order to accurately identify equipment at different distances, in step S1, a high-resolution camera with variable zoom is used to collect images of armed personnel at different distances to construct an equipment detection dataset.

[0054] The collected images are labeled with people and equipment to form an equipment detection dataset, which specifically includes:

[0055] S11. Use annotation boxes to annotate personnel and equipment in the image of armed personnel.

[0056] Specifically, labeling the personnel and equipment in the image is to use a labeling frame to mark the positions of the personnel and equipment in the image and to mark the types of the corresponding equipment.

[0057] S12. Perform dataset enhancement on the annotated image by using annotation box scale distortion, annotation box mirror flipping, random image scaling, random image cropping, and / or random image arrangement to obtain an enhanced dataset. The dataset before enhancement and the enhanced dataset are used as the equipment detection dataset.

[0058] To increase the size of the dataset and improve the model's detection capabilities, data augmentation is performed on the annotated images. Specifically, data augmentation involves scaling the annotated images, mirroring the annotated images, randomizing the image scale, randomizing the image cropping, and / or randomizing the image arrangement to obtain an enhanced dataset.

[0059] During implementation, the scale distortion method is as follows: the coordinate origin of the annotation box remains unchanged, the image in the annotation box is scale-transformed accordingly and then covered on the original target position, the coordinates of the annotation box are transformed along with the scale transformation, and a new image and corresponding annotation information are obtained.

[0060] During implementation, the mirror flipping method is as follows: the coordinate origin of the annotation frame remains unchanged, the image in the annotation frame is flipped and transformed accordingly, and then covered at the original target position, the coordinates of the annotation frame remain unchanged, and a new image and corresponding annotation information are obtained.

[0061] During implementation, armed personnel were equipped with equipment including balaclavas, firearms (including rifles, pistols, and bazookas), explosives, knives, and humans themselves, for a total of five target categories. The final detection dataset included five target categories: humans, balaclavas, firearms, knives, and explosives; the dataset contained over 17,000 real-world images.

[0062] For self-constructed datasets, it is necessary to determine the size of the initial anchor box for the detection model. Therefore, after constructing the equipment detection dataset based on the images of armed personnel and before building the dynamic neural network model, the process also includes determining the initial anchor box for the dynamic neural network model based on the annotated boxes of all images in the dataset.

[0063] Specifically, determining the initial anchor frame of the dynamic neural network model according to the annotated frames of all images in the data set includes:

[0064] S13, scaling each image in the data set to a specified size, to obtain a scaled annotation frame;

[0065] For example, the size of all images in the self-built dataset is 1920×1080; the maximum width and height of each image are proportionally scaled to a specified size, for example, the specified size is 1080×1080, and the smaller sides are also scaled accordingly; the annotation box corresponding to the scaled image is changed from relative coordinates to absolute coordinates, and the size of the scaled annotation box, that is, the length and width of the annotation box, is calculated.

[0066] In order to filter out invalid data, the changed annotation boxes are filtered, and all annotation boxes with a width and height of not less than 2 pixels are retained, and the rest are deleted.

[0067] S14. Cluster the scaled annotation boxes, use a genetic algorithm to mutate each type of annotation box, and select the optimal annotation box as the initial anchor box based on the prediction accuracy.

[0068] For clustering of all the scaled annotation frames, during implementation, the k-means clustering algorithm can be used to cluster the annotation frames to obtain annotation frames of different km categories.

[0069] In object recognition tasks, since large feature maps contain more information about small objects, it is often desirable to detect small objects within them. Therefore, anchor boxes on large feature maps are typically assigned small values, while anchor boxes on small feature maps are assigned larger values. In practice, if a dynamic neural network includes a four-class classifier, four sets of initial anchor boxes are required, each containing three pairs of values. Therefore, in the k-means clustering algorithm, the number of classes km is 12.

[0070] For each class of labeled boxes, a genetic algorithm is used to mutate them. The optimal one is selected as the initial anchor box for that class based on the fitness function. Specifically, the labeled boxes are randomly mutated, i.e., their length and width are randomly mutated to generate a cluster of anchor boxes corresponding to that class. For each of the generated clusters of anchor boxes, the prediction accuracy is calculated, and the anchor box with the highest prediction accuracy is selected as the initial anchor box for that class.

[0071] The prediction accuracy of each anchor box is calculated using the following formula: where p i,j represents the prediction accuracy of the j-th anchor box of the i-th category, n i,j Indicates the number of times the intersection-over-union ratio of the j-th anchor box of the i-th category and the i-th label box is greater than the threshold, m i Represents the number of annotation boxes of the i-th category.

[0072] Specifically, the equipment detection model in step S1 is a dynamic neural network model; the dynamic neural network model includes a first subnetwork and a second subnetwork, the first subnetwork is used to detect humans in the image; when the first subnetwork detects that the image contains humans, the human ROI is extracted and transmitted to the second subnetwork; the second subnetwork is used to perform equipment detection using classifiers at different network depths according to different image resolutions; the dynamic neural network model is trained based on the equipment detection dataset to obtain a trained armed personnel equipment detection model.

[0073] During implementation, the first sub-network can utilize the YOLOv3-tiny architecture, which is lightweight, easy to deploy, and highly accurate. It is suitable for simple person detection tasks. If the first sub-network detects a person in the image, it crops the human region of interest (ROI) and transmits the human ROI to the second sub-network. Otherwise, forward propagation is terminated, thereby improving the model's detection efficiency.

[0074] The second sub-network is used to perform equipment detection using classifiers at different network depths according to different image resolutions, that is, equipment detection is performed at different network depths according to the resolution of the human ROI received from the first sub-network.

[0075] During implementation, the second sub-network can use an improved yolov5l model. The second sub-network includes a backbone network unit, a neck network unit, and a prediction unit. The backbone network unit is used to extract feature maps of different scales of the human ROI. The neck network is used to upsample and fuse the feature maps of different scales extracted by the backbone network unit to obtain tensor data of different scales. The prediction unit includes multiple shallow classifiers for performing target detection based on tensor data of different scales.

[0076] The Neck network unit includes multi-level CSP and CBL combination blocks, and shallow classifiers are connected after the CSP and CBL combination blocks of different levels. The shallow classifiers are used to perform target detection according to tensor data of the current scale.

[0077] To facilitate identification and detection of samples of varying resolutions, shallow classifiers at varying network depths (corresponding to different levels of CSP and CBL blocks) are employed, thereby improving model efficiency and reducing computational redundancy. Specifically, the Neck network unit includes multiple levels of CSP and CBL blocks. The higher the level, the deeper the network. By connecting multiple classifiers of the detection unit to CSP and CBL blocks at different levels, rapid identification and detection of samples of varying resolutions can be achieved, significantly reducing computational effort and conserving computing resources.

[0078] The CBL block includes a fully connected layer, a batch normalization layer, and a leaky ReLU layer. The CSP block is a CSP2_x structure block that splits the input into two branches. One branch first passes through the CBL, then through x residual structures, and then undergoes a convolution. The other branch directly undergoes convolution. The two branches are then concatenated and output after passing through the BN layer and activation layer.

[0079] For example, the classifiers are connected to the last four CSP and CBL combination blocks of the Neck network unit. For ease of description, they are represented as the first shallow classifier, the second shallow classifier, the third shallow classifier, and the fourth shallow classifier. The output of each CSP and CBL combination block is divided into two paths. One path is connected to the shallow classifier corresponding to that level, and the other path continues to forward propagate through the CBL layer and the concat layer in sequence and enters the next level CSP and CBL combination block. The output of the CSP and CBL combination block is first judged for resolution. If the resolution is within the preset range, the feature map is input into the shallow classifier corresponding to the current level for detection and recognition, and the forward propagation is stopped. Otherwise, the feature map continues to forward propagate to further extract deep features. For example, after the fourth-to-last CSP and CBL combination block, the resolution of the input image is first judged. If the resolution is greater than or equal to 400, the feature map is input into the first shallow classifier for detection and recognition; if the image resolution is less than 400, the forward propagation continues to extract deep features. After the third-to-last CSP and CBL block, the input image resolution is first determined. If the resolution is less than 400 and greater than or equal to 200, the feature map is fed into the second shallow classifier for detection and recognition; otherwise, forward propagation continues to extract deeper features. After the second-to-last CSP and CBL block, the input image resolution is first determined. If the resolution is less than 200 and greater than or equal to 50, the feature map is fed into the third shallow classifier for detection and recognition; otherwise, forward propagation continues. After the first-to-last CSP and CBL block, no further determination is required; the features are directly fed into the corresponding classifier for detection and recognition. By connecting classifiers at different depths, samples with different resolutions are retired at different depths, and only samples with resolution less than 50 enter the deep layers of the network for calculation. This greatly improves the model's operational efficiency and reduces computing power redundancy.

[0080] Specifically, the shallow classifier includes a convolution layer, a concat layer and a sigmoid layer connected in sequence; the convolution layer is used to extract image features, the concat layer is used to splice features, and the sigmoid layer is used for classification; the convolution kernel size and number of shallow classifiers at different levels are different.

[0081] In implementation, the convolutional layer of the first shallow classifier consists of 128 convolution kernels, each with a kernel size of 1 and a stride of 1. The convolutional layer of the second shallow classifier consists of 128 convolution kernels, each with a kernel size of 3 and a stride of 2. The convolutional layer of the third shallow classifier consists of 256 convolution kernels, each with a kernel size of 3 and a stride of 2. The convolutional layer of the fourth shallow classifier consists of 384 convolution kernels, each with a kernel size of 3 and a stride of 2.

[0082] During implementation, the concat layer of the shallow classifier is used to concatenate the features output by the convolutional layer with the features output by a layer in the backbone network. This allows the features extracted by the backbone network to be integrated with the features extracted by the neck network, resulting in more accurate classification. During implementation, a layer in the backbone network whose output feature dimensions match those of the convolutional layer of the shallow classifier is selected and its output features are concatenated with the features output by the convolutional layer of the current classifier.

[0083] By setting convolution kernels of different sizes in classifiers of different depths, features of different depths can be extracted. For images with higher resolution, accurate detection and recognition can be performed without extracting deep features, thereby reducing the amount of calculation and improving detection efficiency.

[0084] After the dynamic neural network model is constructed, it is trained based on the equipment detection dataset to obtain a trained armed personnel equipment detection model. Specifically, during model training, the detection box loss of the model is calculated using the following formula:

[0085]

[0086] Among them, dis_2 represents the Euclidean distance between the center point of the predicted box and the labeled box, dis_C represents the diagonal distance between the minimum circumscribed rectangle of the predicted box and the labeled box, and IOU represents the intersection-over-union ratio of the predicted box and the labeled box.

[0087] The classification loss of the model uses the cross entropy loss function:

[0088]

[0089] Where M represents the number of categories. For example, if four types of equipment are to be identified, then M = 4. ic Indicates the confidence that sample i belongs to category c, y ic It is a 0-1 variable that takes 1 when the true category of sample i is c, otherwise it takes 0. N represents the number of samples in a batch.

[0090] The overall loss function of the model is finally obtained:

[0091]

[0092] According to the overall loss of the model, back propagation is performed to optimize the model parameters, thereby obtaining a trained armed personnel equipment detection model.

[0093] After training, equipment recognition models contain a large number of invalid, inefficient, redundant structures and parameters, making further improving inference efficiency challenging. Pruning is one method for improving inference efficiency. By trimming inefficient branches and parameters, it can efficiently generate smaller models with higher memory utilization, lower energy consumption, faster inference speed, and minimal loss in inference accuracy.

[0094] By adopting a dynamic neural network model to construct an armed personnel equipment detection model, different network depths can be used for recognition of images with different resolutions, thereby reducing redundant calculations and improving detection efficiency.

[0095] Specifically, after training the dynamic neural network model based on the equipment detection dataset to obtain a trained armed personnel equipment detection model, the method further includes pruning the armed personnel equipment detection model by performing the following steps:

[0096] S15, performing a sparse processing on each channel of the model and calculating a scale factor of each channel after the sparse processing;

[0097] Because different channels in the network have different effects on model recognition, the purpose of sparsification is to bring the coefficients (also called scale factors) of the BN layers of channels with less influence close to 0, thereby obtaining sparse scale factors. During training, a regularization term is added to the scale factors of the BN layers in each channel. After training, scale factors with smaller values ​​are brought closer to 0, thus achieving the purpose of sparsifying the scale factors.

[0098] S16. If the scale factor is less than a preset threshold, the channel is pruned; otherwise, the channel is retained.

[0099] For example, if the pruning percentage is set to 55%, 55% of the channels will be pruned. Based on the percentage and all the scale factors in the model, a threshold is determined, and all channels with scale factors less than the threshold are pruned (i.e., their corresponding scale factors are set to 0), thereby pruning the model.

[0100] S17. Retrain the pruned model to obtain a trained armed personnel equipment detection model.

[0101] After pruning, the model's recognition accuracy will inevitably decline due to the reduction in the number of channels and model parameters. Therefore, further training and fine-tuning are required to compensate for the accuracy loss caused by pruning. When the accuracy reaches the preset value, training ends, resulting in the pruned armed personnel equipment detection model.

[0102] Since no dataset on armed personnel behavior exists, we need to construct our own dataset. To implement this, we record videos of armed personnel behavior at a 30fps frame rate and a 640×480 resolution to obtain the video stream data. Based on this video stream data, we manually annotate the video frame by frame.

[0103] Specifically, in step S2, constructing a training sample set for armed personnel behavior recognition based on the video stream data includes:

[0104] S21, extracting the skeleton joint point data of each frame image in the video stream data; adding a behavior label to each frame image; matching the behavior label with the skeleton joint point data to obtain an initial training sample set;

[0105] Specifically, we manually add the armed personnel behavior labels to each frame of the image. The armed personnel behavior is divided into 6 categories (see Appendix Figure 3 ): standing, walking, squatting, standing up, standing shooting, squatting shooting. To standardize the annotation results, the shooting angle is constrained to be ±30° horizontal shooting, and the behavior that meets the shooting angle is defined as shooting behavior.

[0106] Use posture recognition algorithm to extract the skeleton joint data of the personnel in the image. When implementing, the AlphaPose posture recognition algorithm can be used to extract the skeleton joint data of the armed personnel in the video frame by frame (see attached Figure 4 ), the skeletal joint data includes the skeletal joint coordinates and confidence scores. A total of 14 skeletal joints are extracted: face center, neck, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right ankle, and left ankle.

[0107] The skeletal joint data of each frame is mapped to the behavioral label to generate the initial training sample set. Since the posture recognition algorithm may contain empty data, the empty data is removed, that is, the image frames without skeletal attachment point data are removed to ensure the validity of the data.

[0108] S22: Perform label smoothing on the behavior labels in the initial training sample set to obtain a behavior recognition training sample set.

[0109] For multi-classification models, labels are usually encoded in one-hot format. For example, the first class label "standing" is encoded as (1, 0, 0, 0, 0, 0), and so on. Using one-hot labels cannot guarantee the generalization ability of the model and makes the network prone to overfitting. To solve this problem, after obtaining the initial training sample set, the behavior labels in the initial sample set are smoothed to obtain the behavior recognition training sample set. Specifically, label smoothing includes:

[0110] S221, overall smoothing of all behavior labels in the initial training sample set

[0111] Specifically, the following formula is used to perform overall smoothing on all behavior labels in the initial training sample set:

[0112]

[0113] Among them, Label represents the sample behavior label, represents the number of categories, and ε represents the smoothing parameter.

[0114] In practice, ε can be set to 0.1. After smoothing the overall behavior labels, a small amount of probability is assigned to classes with lower probabilities, leaving some room for generalization in learning. For example, the label (1, 0, 0, 0, 0, 0) becomes (0.90, 0.02, 0.02, 0.02, 0.02, 0.02) after smoothing.

[0115] S222, determining a behavior transition frame in the initial training sample set, and performing intra-group behavior label smoothing on a group of images before the behavior transition frame;

[0116] Specifically, if the behavior label of the i-1th frame is different from the behavior label of the i-th frame, then the i-th frame is the behavior transition frame. For each behavior transition frame, the group of images before it is smoothed within the group. Specifically, it includes:

[0117] For each behavior conversion frame, determine the active index and target index of the image group before conversion according to the label value of the image frame before the behavior conversion frame and the label value of the behavior conversion frame;

[0118] According to the formula Labels[j][active index] = label maximum value * (ij) / k, calculate the label value corresponding to the active index in the labels of the k images before the behavior conversion frame;

[0119]

[0120] Calculate the label value corresponding to the target index in the labels of the k images before the behavior conversion frame;

[0121] Where j = ik, i-(k-1),...i-1, the i-th frame is the behavior transition frame, Labels[j][active index] represents the label value corresponding to the active index in the behavior label of the j-th frame image; Labels[j][target index] represents the label value corresponding to the target index in the behavior label of the j-th frame image, the active index is the index of the maximum value of the label in the behavior label of the i-1-th frame image, and the target index is the index of the maximum value of the label in the behavior label of the i-th frame image.

[0122] Specifically, with Figure 5 The process of smoothing behavior labels within a group is explained using the behavior labels in as an example. Figure 5 The last row of labels in the image represents the behavior transition frame, assuming it is the i-th frame. A group of k behavior labels preceding the i-th frame is extracted for intra-group label smoothing. During implementation, k can be determined based on the duration of the action and the required smoothing accuracy. A portion of the continuous behavior labels preceding the behavior transition frame is used for intra-group smoothing. For example, if k is 7, intra-group smoothing is performed on the seven behavior labels preceding the i-th frame. In the behavior labels of the i-1-th frame, the index with the maximum value is 0, so the active index is 0. In the behavior labels of the i-th frame, the index with the maximum value is 1, so the target index is 1.

[0123] Therefore, for the j-th frame image, j = ik, i-(k-1), ... i-1, the label value at index 0 is calculated according to Labels[j][active index] = label maximum value * (ij) / k, and according to Calculate the label value at index 1, and the label values ​​at other index positions remain unchanged.

[0124] For example, for the label (0.90, 0.02, 0.02, 0.02, 0.02, 0.02), the maximum label value is 0.9 and the minimum label value is 0.2. The label value after smoothing within the group is as follows Figure 6 As shown in the figure, since human behavior changes continuously, by smoothing the behavior label before the label conversion frame, the behavior label can be smoothly transitioned to the behavior conversion frame, and ultimately the label can better reflect the actual behavior of the person, enhancing the generalization space of subsequent learning and providing a data basis for accurately identifying the actions of armed personnel.

[0125] S223. The skeleton joint point data includes the confidence of the skeleton joint points; confidence smoothing is performed on the behavior label of each image based on the confidence of the skeleton joint points to obtain a behavior recognition training sample set.

[0126] Specifically, for each image frame, the confidence of the main skeletal joints is set to 1, the confidence of other skeletal joints remains unchanged, and the mean confidence of all skeletal joints is calculated; among them, the main skeletal joints include neck, left shoulder, right shoulder, left hip, and right hip.

[0127] The mean of the confidence level is multiplied by the label value of the frame image to obtain the smoothed label of the frame image based on the confidence level. This further enhances the generalization space of learning. The behavior label data after confidence level smoothing is as follows: Figure 7 shown.

[0128] After smoothing the behavior labels, the implementation also includes normalizing the skeletal joint point coordinate data in the initial training sample set. Specifically, the skeletal joint point coordinates can be normalized using the maximum / minimum values ​​of each group of skeletal joint point coordinates, normalizing all skeletal joint point coordinates to the range of (-1, 1).

[0129] During implementation, the training sample set constructed by this application contains a total of 29,757 frames of annotated data for training.

[0130] Specifically, in step S2, the armed personnel behavior recognition model is a model built based on a spatiotemporal graph convolutional network, specifically including:

[0131] S23, constructing a spatiotemporal graph of skeletal joints using skeletal joints as nodes, natural connections between skeletal joints as spatial edges, and connections between the same skeletal joints in two consecutive frames as temporal edges;

[0132] The sequence of human skeleton points can usually be represented by the coordinates of the human joints in each frame of the image. In order to better utilize graph convolution to extract the dynamic information of human skeleton points, the edges between the nodes of the graph should not only contain spatial edges representing the natural connections between human joints, but also temporal edges connecting the same joint points in consecutive time steps, extending the traditional graph convolution to the temporal neighborhood. The constructed spatiotemporal graph of skeleton joint points is as follows: Figure 8 shown.

[0133] The structure of the human skeleton spatiotemporal graph is G = (V, E). The skeletal joints are connected as nodes of the spatiotemporal graph through spatial edges and temporal edges. The information of the spatiotemporal graph includes the number of skeletal joints N, the number of frames T in the input video stream, and the feature matrix v corresponding to each joint. ti The feature matrix of all relevant nodes in the spatiotemporal graph can be expressed as follows:

[0134] V={v ti |t=1,2,KT,i=1,2,KN}

[0135] Among them, v ti The feature matrix of the i-th joint point in the t-th frame contains the coordinates and confidence of the joint point. The nodes in the spatiotemporal graph are connected by spatial edges and temporal edges, which are represented as follows:

[0136] E s ={v ti ,v tj |(i,j)∈H}

[0137] E t ={v ti v (t+1)i}

[0138] Where H is the set of naturally connected joints of the human body. By constructing a spatiotemporal graph of human skeletal joints, we describe the trajectory information of human behavior changing over time.

[0139] S24. Construct a spatiotemporal graph convolutional neural network, wherein the spatiotemporal graph convolutional network comprises a plurality of sequentially connected spatiotemporal graph convolutional blocks;

[0140] Each of the spatiotemporal graph convolution blocks includes a spatial graph convolution layer and a temporal graph convolution layer connected in sequence; the spatial graph convolution layer is used to perform graph convolution on the input features to extract the spatial domain features of the spatiotemporal graph of the skeletal joints; the temporal graph convolution layer is used to perform standard two-dimensional convolution on the input features to extract the temporal domain features of the spatiotemporal graph of the skeletal joints;

[0141] Specifically, the spatial graph convolution layer is used to perform graph convolution on the input features to extract the spatial domain features of the spatiotemporal graph of the skeletal joints, including:

[0142] The neighborhood of each node in the spatiotemporal graph of skeletal joints is partitioned into subsets using a distance-based partitioning method; the adjacency matrix of each node is constructed based on the partitioned subsets;

[0143] In traditional convolutional neural networks, the sampling function can be understood as the size of the convolution kernel, that is, the range covered during each convolution operation (feature extraction). For example, when a 3*3 convolution kernel is convolved on a single pixel, it actually calculates and aggregates the information of that pixel and its eight adjacent pixels.

[0144] In the spatiotemporal graph convolutional network, a node is equivalent to an image pixel point of a traditional convolution, and the sampling function is responsible for specifying the range of adjacent nodes involved when performing graph convolution operations on each node. The present application uses a distance-based partitioning method to subset the neighborhood of each node in the spatiotemporal graph of skeletal joints. In the present application, the neighborhood set is divided into two subsets based on the first-order adjacent nodes (directly connected nodes): 1) d = 0 represents the root node; 2) d = 1 represents the neighborhood subset with a distance of 1 from the root node. Therefore, the number of subsets divided in the present invention is K = 2, and there are two types of weight functions correspondingly. The process of mapping the points in the neighborhood to the subsets after division so that they have the same label can be expressed as: l ti :B(v ti )→{0,1,KK-1}, then the weight function w can be expressed as w(v tj ,v ti )=w(l ti (v tj )). B(v ti ) represents the set of adjacent nodes of the i-th joint point in the t-th frame, l tiRepresents the subset labels of adjacent nodes. The connections between human skeleton joints in a single frame can be expressed as the adjacency matrix A, and the identity matrix I represents the self-connection. For the partitioning strategy based on joint distance, the adjacency matrix will be decomposed into several matrices A j ,have j = 0, 1. In the distance-based partitioning strategy: A0 = I, A1 = A.

[0145] The constructed spatiotemporal graph convolutional network includes multiple sequentially connected spatiotemporal graph convolution blocks, for example, 9 sequentially connected spatiotemporal graph convolution blocks. The first three spatiotemporal graph convolution blocks have 64 channels for output, the next three spatiotemporal graph convolution blocks have 128 channels for output, and the last three spatiotemporal graph convolution blocks have 256 channels for output.

[0146] Among them, each spatiotemporal graph convolution block includes a spatial graph convolution layer and a temporal graph convolution layer connected in sequence.

[0147] The spatial graph convolution layer is based on the formula Perform graph convolution operation to extract spatial features; where f in represents the input features of the spatial graph convolution layer, f out represents the output features of the spatial graph convolution layer, Α j is the adjacency matrix representation of the jth subset, Λ j is the degree matrix of the adjacency matrix of the jth subset, W j represents the weight of the j-th subset, represents the importance mask matrix of the node, Represents bitwise multiplication.

[0148] When the human body is moving, some joints often move in groups (such as wrists and elbows) and may appear in various parts of the body. Therefore, the modeling of these joints should include different importance. Therefore, this application adds a learnable mask in each spatial graph convolution layer. It measures the contribution of a node feature to its neighboring nodes based on the importance weights learned from the edge information in the spatiotemporal graph of the skeletal joints. That is, the spatial graph convolution layer includes an importance mask unit to adaptively adjust the importance of each node to other adjacent nodes.

[0149] The importance mask unit includes a batch normalization layer, a Relu layer, a dropout layer, a convolutional layer and a Sigmoid layer connected in sequence;

[0150] The normalization layer is used to make the importance mask matrix asymmetric; the Relu layer is used for nonlinear transformation; the dropout layer is used to prevent overfitting; the convolution kernel of the convolution layer is 1×1, which is used to make the mask matrix consistent with the dimension of the corresponding graph convolution layer; the Sigmoid layer is used to map the output result to the range of [0,1].

[0151] The temporal graph convolution layer is used to perform standard two-dimensional convolution on the input features to extract the temporal features of the spatiotemporal graph of the skeletal joints. ti The neighborhood of is expanded to include time-connected nodes, which can be expressed as follows The parameter Γ controls the time span in the neighborhood graph, which is called the time kernel size. Since the time axis is ordered, the constructed label mapping function can be modified to the following mapping function: v tj Represents the feature matrix of the j-th joint point in the t-th frame, v qj Represents the feature matrix of the j-th joint point in the q-th frame.

[0152] After constructing the spatiotemporal graph convolutional network, the spatiotemporal graph convolutional network is trained based on the behavior recognition training sample set obtained in step S22 to obtain a trained armed personnel behavior recognition model.

[0153] During implementation, the batch_size can be set to 32, and training is performed for 30 epochs. The loss function uses the BCE loss, and the optimizer uses Adadelta. The initial learning rate is set to 0.01, and the learning rate is multiplied by 0.1 every 10 epochs. The trained armed personnel behavior recognition model achieved an accuracy of 99.2% for armed personnel behavior recognition on the constructed dataset.

[0154] The present invention constructs a training dataset based on video stream data of armed personnel behavior, smoothes the labels in the dataset to provide training data with a certain generalization space for the training model, thereby preventing overfitting of the training model; by adopting a spatiotemporal graph convolutional network to construct an armed personnel behavior recognition model, features are extracted from the time domain and spatial domain, thereby extracting richer and deeper features and improving the accuracy of behavior recognition.

[0155] Specifically, after the equipment detection model and the armed personnel behavior recognition model are trained, step S3 is executed. For each frame of the video stream to be identified, each frame is extracted and input into the trained equipment detection model to obtain an armed personnel equipment detection result for each frame. Skeletal key point data of the armed personnel in each frame of the video stream to be identified is extracted and input into the trained armed personnel behavior recognition model to obtain an armed personnel behavior recognition result for each frame of the video stream.

[0156] In step S3, based on the equipment detection results and behavior recognition results, the risk factor of the person in each frame image is calculated in the following way:

[0157] Danger=P i IoU i +P act , where P i Indicates the confidence of the detected i-th equipment, IoU i represents the intersection-over-union ratio of the detected i-th equipment and human ROI, P act Indicates the dangerousness of the behavior recognition result.

[0158] Specifically, the risk level of the behavior recognition result is calculated according to the following formula:

[0159] Among them, P j represents the confidence of the j-th behavior category, β j represents the risk factor of the jth behavior category, Indicates the number of behavior categories.

[0160] During implementation, the risk level of the behavior recognition result may also be the risk coefficient corresponding to the behavior type with the highest probability obtained by behavior recognition classification.

[0161] The specific risk factor for each behavior category can be set according to the risk of each behavior.

[0162] By integrating equipment characteristics and behavioral characteristics, armed personnel can be judged from two aspects, ultimately achieving efficient and accurate identification of armed personnel.

[0163] One embodiment of the present invention discloses an armed personnel identification system that integrates equipment and behavior characteristics, such as Figure 2 As shown, it includes the following modules:

[0164] An equipment detection model training module is used to obtain an equipment detection data set and train an equipment detection model based on the equipment detection data set;

[0165] A behavior recognition model training module is used to obtain video stream data of armed personnel behavior, construct an armed personnel behavior recognition training sample set based on the video stream data, and train an armed personnel behavior recognition model based on the armed personnel behavior recognition training sample set;

[0166] The armed personnel identification module is used to extract each frame of the video stream to be identified and input it into a trained equipment detection model to obtain an equipment detection result for each frame of the image; extract the skeletal joint point data of each frame of the video stream to be identified; input the said skeletal joint point data into a trained armed personnel behavior recognition model to obtain a personnel behavior recognition result for each frame of the video stream; based on the said equipment detection result and behavior recognition result, calculate the risk factor of the person in each frame of the image; if the risk factor is higher than a preset threshold, the person is judged to be an armed person.

[0167] The above method embodiment and system embodiment are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process can be found in the above embodiment and will not be repeated here.

[0168] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0169] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for identifying armed personnel by integrating equipment and behavioral characteristics, characterized in that: The following steps are involved: Acquire an equipment detection dataset, and train an equipment detection model based on the equipment detection dataset; Obtaining video stream data of armed personnel behavior, and constructing a training sample set for armed personnel behavior recognition based on the video stream data; Training an armed personnel behavior recognition model based on the armed personnel behavior recognition training sample set; Extract each frame of the video stream to be identified and input it into the trained equipment detection model to obtain the equipment detection results of each frame of the image; extract the skeletal joint point data of each frame of the video stream to be identified; Inputting the skeleton joint point data into a trained armed personnel behavior recognition model to obtain personnel behavior recognition results for each frame of the video stream; Based on the equipment detection results and behavior recognition results, the risk factor of the person in each frame of the image is calculated. If the risk factor is higher than a preset threshold, the person is determined to be an armed person; Constructing an armed personnel behavior recognition training sample set based on the video stream data, including: Extracting skeletal joint point data of each frame of the video stream data; adding a behavior label to each frame of the image; and matching the behavior label with the skeletal joint point data to obtain an initial training sample set; Perform label smoothing on the behavior labels in the initial training sample set to obtain a behavior recognition training sample set; Performing label smoothing on the behavior labels in the initial training sample set to obtain a behavior recognition training sample set, including: Perform overall smoothing on all behavior labels in the initial training sample set; Determine the behavior transition frame in the initial training sample set, and perform intra-group behavior label smoothing on a group of images before the behavior transition frame; The skeleton joint point data includes the confidence of the skeleton joint points; based on the confidence of the skeleton joint points, the behavior label of each image is confidence smoothed to obtain a behavior recognition training sample set; The following formula is used to perform overall smoothing on all behavior labels in the initial training sample set: Among them, Label represents the sample behavior label, represents the number of classifications, ε represents the smoothing parameter; Perform intra-group behavior label smoothing on a group of images before the behavior conversion frame, including: For each behavior conversion frame, determine the active index and target index of the image group before conversion according to the label value of the image frame before the behavior conversion frame and the label value of the behavior conversion frame; According to the formula Labels[j][active index] = label maximum value * (ij) / k, calculate the label value corresponding to the active index in the labels of the k images before the behavior conversion frame; According to the formula Calculate the label value corresponding to the target index in the labels of the k images before the behavior conversion frame; Where j = ik, i-(k-1),...i-1, the i-th frame is the behavior transition frame, Labels[j][active index] represents the label value corresponding to the active index in the behavior label of the j-th frame image; Labels[j][target index] represents the label value corresponding to the target index in the behavior label of the j-th frame image, the active index is the index of the maximum value of the label in the behavior label of the i-1-th frame image, and the target index is the index of the maximum value of the label in the behavior label of the i-th frame image.

2. The method for identifying armed personnel by integrating equipment and behavior characteristics according to claim 1, characterized in that: The risk factor of each person in each frame is calculated as follows: Danger=P i IoU i +P act , where P i Indicates the confidence of the detected i-th equipment, IoU i represents the intersection-over-union ratio of the detected i-th equipment and human ROI, P act Indicates the dangerousness of the behavior recognition result.

3. The method for identifying armed personnel by integrating equipment and behavioral characteristics according to claim 2, characterized in that: The dangerousness of the behavior recognition result is calculated according to the following formula: Among them, P j represents the confidence of the j-th behavior category, β j represents the risk factor of the jth behavior category, Indicates the number of behavior categories.

4. The method for identifying armed personnel by integrating equipment and behavioral characteristics according to claim 1, characterized in that: The equipment detection model is a dynamic neural network model; the dynamic neural network model includes a first subnetwork and a second subnetwork, the first subnetwork is used to detect humans in the image; when the first subnetwork detects that the image contains humans, the human ROI is extracted and transmitted to the second subnetwork; the second subnetwork is used to perform equipment detection using classifiers at different network depths according to different image resolutions; the dynamic neural network model is trained based on the equipment detection dataset to obtain a trained armed personnel equipment detection model.

5. The method for identifying armed personnel behavior according to claim 1, characterized in that: The confidence smoothing of the behavior label of each image is performed based on the confidence of the skeleton joint points, including: For each image frame, the confidence of the main skeletal joints is set to 1, the confidence of other skeletal joints remains unchanged, and the mean confidence of all skeletal joints is calculated; The mean of the confidence levels is multiplied by the label value of the frame image to obtain a smoothed label of the frame image based on the confidence levels.

6. An armed personnel identification system integrating equipment and behavioral characteristics, characterized by: Includes the following modules: An equipment detection model training module is used to obtain an equipment detection data set and train an equipment detection model based on the equipment detection data set; A behavior recognition model training module is used to obtain video stream data of armed personnel behavior and construct a training sample set for armed personnel behavior recognition based on the video stream data; Training an armed personnel behavior recognition model based on the armed personnel behavior recognition training sample set; The armed personnel identification module is used to extract each frame of the video stream to be identified and input it into the trained equipment detection model to obtain the equipment detection results of each frame of the image; and extract the skeletal joint point data of each frame of the video stream to be identified; Inputting the skeleton joint point data into a trained armed personnel behavior recognition model to obtain personnel behavior recognition results for each frame of the video stream; Based on the equipment detection results and behavior recognition results, the risk factor of the person in each frame of the image is calculated. If the risk factor is higher than a preset threshold, the person is determined to be an armed person; Constructing an armed personnel behavior recognition training sample set based on the video stream data, including: Extracting skeletal joint point data of each frame of the video stream data; adding a behavior label to each frame of the image; and matching the behavior label with the skeletal joint point data to obtain an initial training sample set; Perform label smoothing on the behavior labels in the initial training sample set to obtain a behavior recognition training sample set; Performing label smoothing on the behavior labels in the initial training sample set to obtain a behavior recognition training sample set, including: Perform overall smoothing on all behavior labels in the initial training sample set; Determine the behavior transition frame in the initial training sample set, and perform intra-group behavior label smoothing on a group of images before the behavior transition frame; The skeleton joint point data includes the confidence of the skeleton joint points; based on the confidence of the skeleton joint points, the behavior label of each image is confidence smoothed to obtain a behavior recognition training sample set; The following formula is used to perform overall smoothing on all behavior labels in the initial training sample set: Among them, Label represents the sample behavior label, represents the number of classifications, and ε represents the smoothing parameter; Perform intra-group behavior label smoothing on a group of images before the behavior conversion frame, including: For each behavior conversion frame, determine the active index and target index of the image group before conversion according to the label value of the image frame before the behavior conversion frame and the label value of the behavior conversion frame; According to the formula Labels[j][active index] = label maximum value * (ij) / k, calculate the label value corresponding to the active index in the labels of the k images before the behavior conversion frame; According to the formula Calculate the label value corresponding to the target index in the labels of the k images before the behavior conversion frame; Where j = ik, i-(k-1),...i-1, the i-th frame is the behavior transition frame, Labels[j][active index] represents the label value corresponding to the active index in the behavior label of the j-th frame image; Labels[j][target index] represents the label value corresponding to the target index in the behavior label of the j-th frame image, the active index is the index of the maximum value of the label in the behavior label of the i-1-th frame image, and the target index is the index of the maximum value of the label in the behavior label of the i-th frame image.

Images