Suspicious person detection method and device based on video analysis

Abstract

The invention discloses a suspicious person detection method and device based on video analysis. The method comprises the following steps: performing target detection on a video frame through a target detection algorithm to obtain a person target detection result and a suspicious article target detection result; performing front and back frame association on the personnel target detection result through a tracking algorithm to obtain a personnel tracking result; recognizing the personnel tracking result through a behavior recognition algorithm constructed based on a large pre-training model CLIP and transfer learning to obtain a personnel behavior recognition result; and matching the personnel behavior recognition result and the suspicious article target detection result with a preset multi-level suspicious behavior table, evaluating the suspicious level of the behavior according to the matching result, and sending out a corresponding alarm signal. The suspicious person detection precision can be improved, the processing measures are perfected, and a powerful guarantee is provided for public safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of video analysis based on deep learning, and specifically relates to a suspicious person detection method and device based on video analysis. Background Art

[0002] With the increasing demand for public safety, suspicious person detection technology has become one of the key technologies in the security field. Driven by the wave of intelligence and informatization, computer vision technology has developed rapidly. The traditional way of relying on manual monitoring to judge suspicious persons has gradually been replaced by automated detection systems. At present, suspicious person detection solutions based on video analysis technology are emerging in an endless stream, which are mainly divided into the following categories:

[0003] (1) Suspicious person detection based on identity recognition: This method uses face recognition technology to quickly capture and identify individual identities in real-time video, and identifies strangers who appear frequently as suspicious persons;

[0004] (2) Suspicious person detection based on appearance features: This method identifies suspicious persons by judging visual features such as clothing features and facial occlusion degree;

[0005] (3) Suspicious person detection based on behavior recognition: This type of method identifies suspicious persons by judging the behavioral characteristics of the person, mainly including a) action trajectory analysis: by analyzing the person's action trajectory, the person with wandering behavior is judged as a suspicious person; b) key area monitoring: by demarcating key areas, the person who invades the area is classified as a suspicious person; c) behavior pattern recognition: the behavior of the person is classified through the behavior recognition algorithm, and the person with suspicious behavior is judged as a suspicious person.

[0006] Although these methods have reduced the cost of manual monitoring to a certain extent and improved the accuracy and efficiency of detection, there are still some limitations and challenges:

[0007] (1) Low detection accuracy: The existing methods have low detection accuracy and are easily interfered with. Existing suspicious person detection based on identity recognition and suspicious person detection based on appearance features are easily affected by the angle of the person, light changes, and obstructions, resulting in frequent false alarms and missed alarms. Existing suspicious person detection methods based on behavior recognition are mostly limited to simple statistical analysis of behavior patterns, lacking in-depth understanding and analysis of complex behavior patterns, especially when identifying subtle, continuous, and complex behavior patterns. Existing technologies are unable to cope with this.

[0008] (2) Insufficient measures to deal with suspicious persons: Existing suspicious person identification systems mostly rely on preset thresholds or rules for judgment, and lack a dynamic assessment of the degree of suspicion and a graded response mechanism. This results in all suspicious behaviors being treated equally, which not only increases the burden on security personnel, but also reduces the pertinence and efficiency of the response.

[0009] Therefore, although suspicious person detection technology has made remarkable achievements, it still needs to continue to explore and innovate in aspects such as detection accuracy and processing measures to better serve the public security needs of society. Summary of the invention

[0010] In view of the above, the purpose of the present invention is to provide a suspicious person detection method and device based on video analysis, which realizes efficient and accurate detection and tracking of people and suspicious objects by combining target detection algorithm and tracking algorithm, and proposes a behavior recognition algorithm SideActionClip based on large pre-trained model CLIP and transfer learning to accurately recognize and analyze the tracked people, and match the results of people behavior recognition and suspicious object target detection with multi-level suspicious behavior tables to evaluate their suspicious degree, and send out alarm signals of different severity levels. This method performs well in scenarios with high security and high confidentiality requirements, can effectively improve the accuracy of suspicious person detection and further improve suspicious person handling measures, and can provide strong protection for public safety.

[0011] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0012] In a first aspect, an embodiment of the present invention provides a method for detecting suspicious persons based on video analysis, comprising the following steps:

[0013] The target detection algorithm is used to detect the target of the video frame to obtain the human target detection result and the suspicious object target detection result;

[0014] The person tracking result is obtained by associating the previous and next frames of the person target detection result through the tracking algorithm;

[0015] The personnel tracking results are identified by the behavior recognition algorithm SideActionClip built based on the large pre-trained model CLIP and transfer learning to obtain the personnel behavior recognition results, including: inputting the personnel tracking results into the image encoder of the large pre-trained model CLIP after passing through the 2D convolution layer to extract features, inputting the personnel tracking results into the first 3D convolution layer and fusing them with the features extracted by the image encoder, inputting the fused features into the side network module for further feature extraction, and then passing through the average pooling layer, the linear layer and the Softmax layer in sequence to finally obtain the personnel behavior recognition results;

[0016] The results of human behavior recognition and suspicious object target detection are matched with the preset multi-level suspicious behavior table, and the suspicious level of the behavior is evaluated according to the matching results and a corresponding alarm signal is issued.

[0017] Preferably, the target detection algorithm adopts the YOLO model, which includes: a backbone network, a neck network and a head network. First, the video frame is input into the backbone network to extract features, and then the features from different stages of the backbone network are fused through the neck network. Finally, the fused features are sent to the head network to generate personnel target detection results and suspicious object target detection results.

[0018] Preferably, the tracking algorithm adopts the ByteTrack algorithm, which includes: first matching the high-confidence detection box in the personnel target detection result that is greater than or equal to the set threshold with the existing trajectory, and then performing a secondary match on the unmatched trajectory using the low-confidence detection box in the personnel target detection result that is less than the set threshold, and finally obtaining a coherent personnel tracking result through the matching and updating mechanism of the trajectory between consecutive frames.

[0019] Preferably, the image encoder of the large pre-trained model CLIP adopts at least one layer of ViT module built based on Vision Transformer, and the structure of each layer of ViT module includes: a first normalization layer, a first multi-head self-attention module, a second normalization layer and a first multi-layer perceptron module connected in sequence, a first residual connection is established between the input end of the first normalization layer and the output end of the first multi-head self-attention module, and a second residual connection is established between the input end of the second normalization layer and the output end of the first multi-layer perceptron module, and the formula is expressed as follows:

[0020] z l ′=MSA1(LN1(z l-1 ))+z l-1

[0021] z l =MLP1(LN2(z l ′))+z l '

[0022] Among them, LN1(·) and LN2(·) represent the first normalization layer and the second normalization layer respectively, MSA1(·) represents the first multi-head self-attention module, MLP1(·) represents the first multi-layer perceptron module, and z l-1 represents the output of the l-1th layer ViT module. When l is 1, z0 represents the output of the 2D convolutional layer. l ′ represents the output of the first multi-head self-attention module in the l-th layer ViT module and z l-1 The output after the first residual connection, z lRepresents the output of the first multilayer perceptron module in the lth layer ViT module and z l ′The output after the second residual connection, z l That is the output of the l-th layer ViT module.

[0023] Preferably, at least one layer of side network modules is constructed, and the structure of each layer of the side network module includes: a spatiotemporal convolution module, a third normalization layer, a second multi-head self-attention module, a fourth normalization layer and a second multi-layer perceptron module connected in sequence, a third residual connection is established between the input end and the output end of the spatiotemporal convolution module, a fourth residual connection is established between the input end of the third normalization layer and the output end of the second multi-head self-attention module, and a fifth residual connection is established between the input end of the fourth normalization layer and the output end of the second multi-layer perceptron module. The formula is expressed as follows:

[0024] y l ′=y l-1 +z l

[0025] y l = Temporal(y l ′)+y l '

[0026] y l ″′=MSA2(LN3(y l ″))+y l ″

[0027] y l =MLP2(LN4(y l ″′))+y l ″′

[0028] Among them, Temporal(·) represents the spatiotemporal convolution module, LN3(·) and LN4(·) represent the third normalization layer and the fourth normalization layer respectively, MSA2(·) represents the second multi-head self-attention module, MLP2(·) represents the second multi-layer perceptron module, z l represents the output of the ViT module at layer l, y l-1 represents the output of the l-1th layer side network module. When l is 1, y0 represents the output of the first 3D convolutional layer. l ′ represents the input of the network module on the layer l side, y l ″ represents the output of the l-th layer spatiotemporal convolution module and y l ′The output after the third residual connection, y l ″′ represents the output of the second multi-head self-attention module in the lth layer and y l ″The output after the fourth residual connection, y l Represents the output of the second multilayer perceptron module in layer l and yl ″′ is the output after the fifth residual connection, y l That is the output of the network module on the layer 1 side.

[0029] Preferably, the structure of the spatiotemporal convolution module comprises: a second 3D convolution layer, a 2D time adaptive convolution layer and a third 3D convolution layer connected in sequence;

[0030] The structure of the 2D time adaptive convolution layer includes: a spatial global average pooling layer, a temporal global average pooling layer, a fully connected layer, a first 1D convolution layer, a batch normalization layer, a ReLU layer and a second 1D convolution layer connected in sequence. The sixth residual connection is established between the input end of the temporal global average pooling layer and the output end of the fully connected layer. The output of the second 1D convolution layer is used as the calibration weight of the 2D time adaptive convolution layer. The calibration weight is added by 1 and then multiplied by the basic weight of the 2D time adaptive convolution layer to obtain the convolution weight of each frame of the 2D time adaptive convolution layer.

[0031] Preferably, the contents of the multi-level suspicious behavior table include: suspicious levels, suspicious behaviors corresponding to each suspicious level, and suspicious objects corresponding to each suspicious behavior, and a countermeasure description is added to each suspicious level for reference by security personnel.

[0032] In a second aspect, in order to achieve the above-mentioned invention object, an embodiment of the present invention further provides a suspicious person detection device based on video analysis, comprising: a target detection unit, a target tracking unit, a behavior recognition unit and a matching alarm unit;

[0033] The target detection unit is used to perform target detection on the video frame through a target detection algorithm to obtain a person target detection result and a suspicious object target detection result;

[0034] The target tracking unit is used to associate the previous and next frames of the personnel target detection results through a tracking algorithm to obtain a personnel tracking result;

[0035] The behavior recognition unit is used to identify the personnel tracking results through the behavior recognition algorithm based on the large pre-trained model CLIP and transfer learning to obtain the personnel behavior recognition results, including: inputting the personnel tracking results into the image encoder of the large pre-trained model CLIP after passing through the 2D convolution layer to extract features, inputting the personnel tracking results into the first 3D convolution layer and fusing them with the features extracted by the image encoder, inputting the fused features into the side network module to further extract features, and then passing through the average pooling layer, the linear layer and the Softmax layer in sequence to finally obtain the personnel behavior recognition results;

[0036] The matching alarm unit is used to match the personnel behavior recognition results and the suspicious object target detection results with a preset multi-level suspicious behavior table, evaluate the suspicious level of the behavior according to the matching results and send out a corresponding alarm signal.

[0037] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a suspicious person detection device based on video analysis, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned suspicious person detection method based on video analysis when executing the computer program.

[0038] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned suspicious person detection method based on video analysis is implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention performs target detection and person tracking on video frames through a target detection algorithm and a tracking algorithm, and performs suspicious person behavior analysis in combination with a behavior recognition algorithm, thereby avoiding the problem that suspicious person detection methods based on identity recognition and appearance features are easily affected by the angle of the person, light changes, and obstructions, resulting in frequent false alarms and missed alarms, thereby improving the efficiency and accuracy of suspicious person detection.

[0041] (2) The present invention proposes a behavior recognition algorithm SideActionClip based on a large pre-trained model CLIP and transfer learning. By migrating the large pre-trained image model to the video behavior recognition task, it can deeply understand and analyze human behavior, thereby obtaining accurate behavior recognition results and improving the accuracy of suspicious person detection.

[0042] (3) The present invention matches the results of human behavior recognition and suspicious object target detection with a preset multi-level suspicious behavior table, classifies the degree of suspicion, evaluates the suspicious degree of the behavior, and issues an alarm signal of the corresponding severity level. This hierarchical processing method avoids the situation where all suspicious behaviors are treated equally, greatly reduces the burden on security personnel, and at the same time improves the pertinence and efficiency of the response, further improves the suspicious person handling measures, and provides a strong guarantee for public safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 1 is a flow chart of a suspicious person detection method based on video analysis provided by an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of the network structure of the behavior recognition algorithm SideActionClip provided in an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the network structure of a 2D time adaptive convolutional layer in the behavior recognition algorithm provided by an embodiment of the present invention;

[0047] Figure 4 It is a structural schematic diagram of a suspicious person detection device based on video analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0049] The inventive concept of the present invention is: in view of the problems of insufficient accuracy and imperfect handling measures of suspicious person detection methods in the prior art, the embodiments of the present invention provide a suspicious person detection method and device based on video analysis, which performs target detection and person tracking on video frames through a target detection algorithm and a tracking algorithm, and performs suspicious person behavior analysis in combination with a behavior recognition algorithm, wherein the behavior recognition algorithm is constructed based on a large pre-trained model CLIP and transfer learning, and can migrate a large pre-trained image model to a video behavior recognition task, thereby improving the efficiency and accuracy of suspicious person detection, and finally, by matching the person behavior recognition results and the suspicious object target detection results with a preset multi-level suspicious behavior table, the suspicion degree is graded and an alarm signal of the corresponding severity level is issued, thereby improving the pertinence and efficiency of the response, further improving the suspicious person handling measures, and providing a strong guarantee for public safety.

[0050] Figure 1 FIG. 1 is a flow chart of a suspicious person detection method based on video analysis provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a suspicious person detection method based on video analysis, comprising the following steps:

[0051] S1, target detection is performed on the video frame through the target detection algorithm to obtain the human target detection result and the suspicious object target detection result.

[0052] In the embodiment, the real-time video data of the security camera is first read through Python's OpenCV library and the RTSP protocol, and the video frames are read continuously. Then, the images of the video frames are preprocessed including frame skipping and size adjustment. In order to increase the detection speed, the detection is performed in a frame skipping manner, and it is set to detect 1 frame every 3 frames, and the frame image size is adjusted to 640×640 pixels.

[0053] In the embodiment, in order to balance the speed and accuracy of target recognition and realize real-time video analysis, the target detection algorithm adopts the YOLOv8s model with moderate parameters, which includes the following parts:

[0054] (1) Backbone network: Responsible for extracting features from the input image and converting the image into a feature representation with rich semantic information. The backbone network of the YOLOv8s model uses a series of convolutional layers and deconvolutional layers, combined with residual connections and bottleneck structures to reduce the size of the network and improve performance. The C2F module is used as the basic building block. The C2F module reduces redundant parameters through optimized structural design, improves computational efficiency and feature extraction capabilities. In addition, the backbone network of the YOLOv8s model also integrates technologies such as deep separable convolution and dilated convolution to further enhance the feature extraction capability.

[0055] (2) Neck network: It is an intermediate layer that enhances the feature representation capability by fusing feature maps from different stages of the backbone network, and finally passes them to the head network for final detection. The neck network of the YOLOv8s model uses the PAN (Path Aggregation Network) structure for feature fusion and enhancement. PAN is a deep neural network architecture for image semantic segmentation. It implements multi-layer feature aggregation through top-down and bottom-up paths, enhances the feature representation of objects of different scales, and improves detection performance. At the same time, the SPPF (Spatial Pyramid Pooling Fast) module is also used in the neck network for pooling operations of different scales, splicing feature maps of different scales together to improve the detection capability of objects of different sizes.

[0056] (3) Head network: responsible for further processing the features output by the neck network to generate the final human target detection results and suspicious object target detection results. The head network of the YOLOv8s model mainly includes extracting high-level features through several convolutional layers and activation functions, and generating outputs such as predicted bounding boxes, confidence scores, and category probabilities through the prediction layer. Finally, non-maximum suppression (NMS) is used to process the prediction results, remove duplicate detection boxes, and ensure that each target is detected only once.

[0057] (4) Loss function: Two loss functions are used: bounding box regression loss and classification loss. Among them, the bounding box regression loss is used to calculate the difference between the predicted bounding box and the true bounding box. In the embodiment, the bounding box regression loss is calculated using the mean square error (MSE). The classification loss is used to measure the difference between the category distribution predicted by the model and the true label. In the embodiment, the classification loss is calculated using the cross entropy loss.

[0058] S2, using the tracking algorithm to associate the previous and next frames of the person target detection results to obtain the person tracking results.

[0059] In the embodiment, in order to balance the speed and accuracy of target recognition and realize real-time video analysis, the target tracking adopts the high-performance tracking algorithm ByteTrack. Unlike the previous method of only retaining high-confidence detection frames, the ByteTrack algorithm retains almost all detection frames and divides them into high-scoring and low-scoring parts. Retaining low-scoring detection frames can reduce missed detections, maintain trajectory continuity, improve robustness in complex scenes, and ensure that the target can still be tracked when it is occluded or partially visible, thereby significantly improving the overall performance of multi-target tracking.

[0060] The flow of the ByteTrack algorithm is as follows:

[0061] (1) Detection: For each frame, the predicted bounding box D is obtained by the target detection algorithm. k and its confidence score, and D is divided according to the predefined threshold τ k Divided into high-resolution bounding boxes D high and the low score bounding box D low .

[0062] (2) Trajectory prediction: Use the Kalman filter to predict the new position of each trajectory in the existing trajectory T.

[0063] (3) First association: The high-score bounding box D high Match with the predicted trajectory T, use intersection over union (IoU) or re-identification (Re-ID) to calculate the similarity, and complete the matching based on the similarity through the Hungarian algorithm. The unmatched bounding boxes and trajectories are stored in D remain and T remain middle.

[0064] (4) Second association: The remaining trajectory T remain With the low score bounding box D low For matching, only the intersection-over-union ratio is used to calculate the similarity, and after the matching is completed, the unmatched low-scoring bounding boxes are discarded.

[0065] (5) Track management: Delete unmatched tracks after a certain number of frames, and remove them from D remain The new trajectory is initialized in , and finally a coherent person tracking result is obtained through the matching and updating mechanism of the trajectory between consecutive frames.

[0066] S3, the personnel tracking results are identified by the behavior recognition algorithm built based on the large pre-trained model CLIP and transfer learning to obtain the personnel behavior recognition results.

[0067] In the field of computer vision, large pre-trained visual models have powerful feature extraction capabilities. For example, the CLIP model is a model pre-trained on a large-scale image-text pair dataset. It consists of an image encoder and a text encoder. The image encoder adopts the structure of ViT (Vision Transformer). Due to the large size of its pre-training dataset and its own powerful feature extraction capabilities, the CLIP model can learn rich feature representations and complex patterns, and has achieved remarkable results in image tasks. However, in video understanding tasks, due to the limitations of high-quality video datasets and computing resources, pre-training large video models faces challenges. Existing methods usually apply pre-trained image models, such as CLIP, to video tasks after transfer learning, but the computational cost of fine-tuning the full model is high.

[0068] In the embodiment, a behavior recognition algorithm SideActionClip based on a large pre-trained model CLIP and transfer learning is proposed for tracking suspicious behavior recognition of personnel, and solves the problem of high computational cost of fine-tuning the full model. SideActionClip obtains multi-level spatial features from the image encoder of CLIP as a pre-trained model, and fuses these features into an additional lightweight side network module through linear projection and element addition to obtain behavior recognition results. In the process of transfer learning, the weight parameters of the pre-trained CLIP image encoder are frozen so that these parameters do not participate in the update during the training process, so there is no need for back propagation calculation. The traditional adapter method requires splicing the network layers that need to be trained between different layers of the pre-trained model, which results in multiple back propagations and parameter updates between different layers of the pre-trained model during training, which will generate a large number of intermediate calculation results and gradient information, resulting in increased memory usage. The side network module in the embodiment of the present invention avoids these redundant calculations and a large amount of memory consumption by external attachment.

[0069] The network structure of the behavior recognition algorithm SideActionClip is as follows Figure 2 As shown in the figure, the personnel tracking result is input into the 2D convolution layer and then passed through the image encoder of the large pre-trained model CLIP for feature extraction. The personnel tracking result is input into the first 3D convolution layer and fused with the features extracted by the image encoder. The fused features are input into the side network module for further feature extraction and then passed through the average pooling layer, linear layer and Softmax layer in sequence to finally obtain the personnel behavior recognition result.

[0070] The image encoder of the large pre-trained model CLIP uses at least one layer of ViT module (ViT Block) built on Vision Transformer, such as Figure 2 As shown, the structure of each layer of the ViT module includes: a first normalized layer, a first multi-head self-attention module, a second normalized layer and a first multi-layer perceptron module connected in sequence, a first residual connection is established between the input end of the first normalized layer and the output end of the first multi-head self-attention module, and a second residual connection is established between the input end of the second normalized layer and the output end of the first multi-layer perceptron module. The formula is expressed as follows:

[0071] z l ′=MSA1(LN1(z l-1 ))+z l-1

[0072] z l =MLP1(LN2(z l ′))+z l '

[0073] Among them, LN1(·) and LN2(·) represent the first normalization layer and the second normalization layer respectively, MSA1(·) represents the first multi-head self-attention module, MLP1(·) represents the first multi-layer perceptron module, and z l-1 represents the output of the l-1th layer ViT module. When l is 1, z0 represents the output of the 2D convolutional layer. l ′ represents the output of the first multi-head self-attention module in the l-th layer ViT module and z l-1 The output after the first residual connection, z l Represents the output of the first multilayer perceptron module in the lth layer ViT module and z l ′The output after the second residual connection, z l That is the output of the l-th layer ViT module.

[0074] Construct at least one side network module (Side Block). The number of layers of the side network module is the same as the number of layers of the ViT module, such as Figure 2As shown, the structure of each side network module includes: a spatiotemporal convolution module, a third normalization layer, a second multi-head self-attention module, a fourth normalization layer and a second multi-layer perceptron module connected in sequence, a third residual connection is established between the input end and the output end of the spatiotemporal convolution module, a fourth residual connection is established between the input end of the third normalization layer and the output end of the second multi-head self-attention module, and a fifth residual connection is established between the input end of the fourth normalization layer and the output end of the second multi-layer perceptron module. The formula is expressed as follows:

[0075] y l ′=y l-1 +z l

[0076] y l = Temporal(y l ′)+y l '

[0077] y l ″′=MSA2(LN3(y l ″))+y l ″

[0078] y l =MLP2(LN4(y l ″′))+y l ″′

[0079] Among them, Temporal(·) represents the spatiotemporal convolution module, LN3(·) and LN4(·) represent the third normalization layer and the fourth normalization layer respectively, MSA2(·) represents the second multi-head self-attention module, MLP2(·) represents the second multi-layer perceptron module, z l represents the output of the ViT module at layer l, y l-1 represents the output of the l-1th layer side network module. When l is 1, y0 represents the output of the first 3D convolutional layer. l ′ represents the input of the network module on the layer l side, y l ″ represents the output of the l-th layer spatiotemporal convolution module and y l ′The output after the third residual connection, y l ″′ represents the output of the second multi-head self-attention module in the lth layer and y l ″The output after the fourth residual connection, y l Represents the output of the second multilayer perceptron module in layer l and y l ″′ is the output after the fifth residual connection, y l That is the output of the network module on the layer 1 side.

[0080] The lightweight spatiotemporal convolution module is used for spatiotemporal feature extraction of the side network module of the SideActionClip algorithm. The structure of the spatiotemporal convolution module is as follows: Figure 3 As shown, it includes: a second 3D convolution layer, a 2D time adaptive convolution layer and a third 3D convolution layer connected in sequence.

[0081] The 2D time-adaptive convolution layer is a convolution method for video understanding. The core idea is to dynamically calibrate the convolution weights according to the local and global temporal context of each frame, so that the spatial convolution weights are adaptive in the temporal dimension. Figure 3 As shown, the structure of the 2D time adaptive convolution layer includes: a spatial global average pooling layer, a temporal global average pooling layer, a fully connected layer, a first 1D convolution layer, a batch normalization layer, a ReLU layer, and a second 1D convolution layer connected in sequence, a sixth residual connection is established between the input end of the temporal global average pooling layer and the output end of the fully connected layer, the output of the second 1D convolution layer is used as the calibration weight of the 2D time adaptive convolution layer, the calibration weight is added by 1 and then multiplied by the basic weight of the 2D time adaptive convolution layer to obtain the convolution weight of each frame of the 2D time adaptive convolution layer.

[0082] The specific principle and formula of the 2D time adaptive convolution layer are as follows:

[0083] (1) Base weights and calibration weights: The convolution weights W of each frame of the 2D time-adaptive convolution layer t is decomposed into the basic weight W b and the calibration weight α t The product of:

[0084] y t =W t *x t =(α t ·W b )*x t

[0085] Among them, x t represents the input of the 2D time adaptive convolutional layer at time t, y t represents the output of the 2D time-adaptive convolutional layer at time t, W b represents the base weight of the 2D time-adaptive convolutional layer, α t represents the calibration weight of the 2D time-adaptive convolutional layer at time t.

[0086] (2) Calibration weight generation: Calibration weight α t The current frame and its temporal context (i.e. previous and next frames) need to be considered:

[0087] α t =F(…,x t-1 ,xt ,x t+1 ,…)

[0088] The specific implementation of the generating function F(·) is as follows:

[0089]

[0090]

[0091]

[0092] in, represents the feature vector containing the current frame and its neighboring frames, Indicates that by The result of spatial global average pooling, GAP s (·) represents spatial global average pooling, δ(·) represents the ReLU activation function, BN(·) represents batch normalization, r represents the dimension reduction ratio, C represents the number of channels, Conv1D1(·) and Conv1D2(·) represent the first 1D convolutional layer and the second 1D convolutional layer, respectively.

[0093] (3) Introducing global temporal context: In order to enhance the model’s perception of global temporal information, 2D time-adaptive convolution introduces global temporal context into the weight generation process:

[0094] g=GAP t (GAP s (x t ))

[0095]

[0096] Among them, GAP t (·) denotes Temporal Global Average Pooling, so that the descriptor g contains the global temporal context in the input video, and FC(·) denotes the fully connected layer that maps the global descriptor g to the weight generation process.

[0097] (4) Initial weights: In order to utilize the pre-trained weights, the calibration weights α calculated above are t The result is then added with 1, so that the time-adaptive convolution is initialized to the same state as the standard convolution behavior:

[0098]

[0099] This is done in the initial state of the time adaptive convolution α t =1,W t =αt ·W b =W b , which enables loading the trained weights for a standard 2D convolution.

[0100] S4, matching the human behavior recognition results and suspicious object target detection results with a preset multi-level suspicious behavior table, evaluating the suspicious level of the behavior according to the matching results and issuing a corresponding alarm signal.

[0101] In the embodiment, after obtaining the results of human behavior recognition and suspicious object target detection, they will be matched with the preset multi-level suspicious behavior table to evaluate the suspicious degree of the behavior, and an alarm signal of the corresponding severity level will be issued. The multi-level suspicious behavior table is shown in Table 1, in which the normal human behavior category is omitted. The content in Table 1 includes: suspicious level, suspicious behavior corresponding to each suspicious level, and suspicious objects corresponding to each suspicious behavior, and a countermeasure description is added for each suspicious level for reference by security personnel.

[0102] Table 1 Multi-level suspicious behavior table

[0103]

[0104]

[0105] In summary, the suspicious person detection method based on video analysis provided by the embodiment of the present invention can accurately and efficiently identify suspicious persons in scenarios with high security requirements and high confidentiality requirements, effectively improve the suspicious person detection accuracy, further improve the suspicious person handling measures, provide strong protection for public safety, and have significant scientificity and practical application value.

[0106] Based on the same inventive concept, Figure 4 As shown, the embodiment of the present invention further provides a suspicious person detection device 400 based on video analysis, including: a target detection unit 410, a target tracking unit 420, a behavior recognition unit 430 and a matching alarm unit 440.

[0107] The target detection unit 410 is used to perform target detection on the video frame by using a target detection algorithm to obtain a person target detection result and a suspicious object target detection result.

[0108] The target tracking unit 420 is used to associate the previous and next frames of the personnel target detection results through a tracking algorithm to obtain a personnel tracking result.

[0109] The behavior recognition unit 430 is used to identify the personnel tracking results through the behavior recognition algorithm constructed based on the large pre-trained model CLIP and transfer learning to obtain the personnel behavior recognition results, including: inputting the personnel tracking results into the 2D convolution layer and then extracting features through the image encoder of the large pre-trained model CLIP, inputting the personnel tracking results into the first 3D convolution layer and then fusing them with the features extracted by the image encoder, inputting the fused features into the side network module for further feature extraction, and then successively passing through the average pooling layer, the linear layer and the Softmax layer to finally obtain the personnel behavior recognition results.

[0110] The matching alarm unit 440 is used to match the human behavior recognition results and the suspicious object target detection results with a preset multi-level suspicious behavior table, evaluate the suspicious level of the behavior according to the matching results, and issue a corresponding alarm signal.

[0111] Based on the same inventive concept, an embodiment of the present invention also provides a suspicious person detection device based on video analysis, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned suspicious person detection method based on video analysis when executing the computer program.

[0112] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned suspicious person detection method based on video analysis is implemented.

[0113] It should be noted that the suspicious person detection device based on video analysis, the suspicious person detection equipment based on video analysis and the computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the suspicious person detection method based on video analysis. The specific implementation process is detailed in the embodiment of the suspicious person detection method based on video analysis, which will not be repeated here.

[0114] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A suspicious person detection method based on video analysis, characterized in that: The following steps are involved: The target detection algorithm is used to detect the target of the video frame to obtain the human target detection result and the suspicious object target detection result; The person tracking result is obtained by associating the previous and next frames of the person target detection result through the tracking algorithm; The personnel tracking results are identified by a behavior recognition algorithm based on a large pre-trained model CLIP and transfer learning to obtain a personnel behavior recognition result, including: inputting the personnel tracking result into the image encoder of the large pre-trained model CLIP after passing through a 2D convolution layer to extract features, inputting the personnel tracking result into the first 3D convolution layer and fusing it with the features extracted by the image encoder, inputting the fused features into the side network module to further extract features, and then passing through an average pooling layer, a linear layer, and a Softmax layer in sequence to finally obtain a personnel behavior recognition result; The results of human behavior recognition and suspicious object target detection are matched with the preset multi-level suspicious behavior table, and the suspicious level of the behavior is evaluated according to the matching results and a corresponding alarm signal is issued.

2. The method for detecting suspicious persons based on video analysis according to claim 1, characterized in that: The target detection algorithm uses the YOLO model, which includes: backbone network, neck network and head network. First, the video frame is input into the backbone network to extract features, and then the features from different stages of the backbone network are fused through the neck network. Finally, the fused features are sent to the head network to generate human target detection results and suspicious object target detection results.

3. The method for detecting suspicious persons based on video analysis according to claim 1, characterized in that: The tracking algorithm adopts the ByteTrack algorithm. The ByteTrack algorithm includes: first, matching the high-confidence detection boxes in the personnel target detection results that are greater than or equal to the set threshold with the existing trajectories, and then performing secondary matching on the unmatched trajectories using the low-confidence detection boxes in the personnel target detection results that are less than the set threshold. Finally, a coherent personnel tracking result is obtained through the matching and updating mechanism of the trajectories between consecutive frames.

4. The method for detecting suspicious persons based on video analysis according to claim 1, characterized in that: The image encoder of the large pre-trained model CLIP adopts at least one layer of ViT module built based on Vision Transformer. The structure of each layer of ViT module includes: the first normalization layer, the first multi-head self-attention module, the second normalization layer and the first multi-layer perceptron module connected in sequence. The first residual connection is established between the input end of the first normalization layer and the output end of the first multi-head self-attention module, and the second residual connection is established between the input end of the second normalization layer and the output end of the first multi-layer perceptron module. The formula is expressed as follows: with l ′ =MSA1(LN1(with l-1 ))+z l-1 With l =MLP1(LN2(from l ′ ))+z l ′ Among them, LN1(·) and LN2(·) represent the first normalization layer and the second normalization layer respectively, MSA1(·) represents the first multi-head self-attention module, MLP1(·) represents the first multi-layer perceptron module, and z l-1 represents the output of the l-1th layer ViT module. When l is 1, z0 represents the output of the 2D convolutional layer. l ′ represents the output of the first multi-head self-attention module in the l-th layer ViT module and z l-1 The output after the first residual connection, z l Represents the output of the first multilayer perceptron module in the lth layer ViT module and z l ′ The output after the second residual connection, z l That is the output of the l-th layer ViT module.

5. The method for detecting suspicious persons based on video analysis according to claim 1, characterized in that: Construct at least one layer of side network modules, and the structure of each layer of side network modules includes: a spatiotemporal convolution module, a third normalization layer, a second multi-head self-attention module, a fourth normalization layer, and a second multi-layer perceptron module connected in sequence, a third residual connection is established between the input end and the output end of the spatiotemporal convolution module, a fourth residual connection is established between the input end of the third normalization layer and the output end of the second multi-head self-attention module, and a fifth residual connection is established between the input end of the fourth normalization layer and the output end of the second multi-layer perceptron module. The formula is expressed as follows: and l ′ =and l-1 +z l and l "Temporary(and l ′ )+y l ′ <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> ″′=MSA2(LN3(y<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> ″))+y<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> ″ and l =MLP2(LN4(and l ″′))+y l " Among them, Temporal(·) represents the spatiotemporal convolution module, LN3(·) and LN4(·) represent the third normalization layer and the fourth normalization layer respectively, MSA2(·) represents the second multi-head self-attention module, MLP2(·) represents the second multi-layer perceptron module, z l represents the output of the ViT module at layer l, y l-1 represents the output of the l-1th layer side network module. When l is 1, y0 represents the output of the first 3D convolutional layer. l ′ represents the input of the network module on the layer l side, y l ″ represents the output of the l-th layer spatiotemporal convolution module and y l ′ The output after the third residual connection, y l ″′ represents the output of the second multi-head self-attention module in the lth layer and y l ″The output after the fourth residual connection, y l Represents the output of the second multilayer perceptron module in layer l and y l ″′ is the output after the fifth residual connection, y l That is the output of the network module on the layer 1 side.

6. The method for detecting suspicious persons based on video analysis according to claim 5, characterized in that: The structure of the spatiotemporal convolution module includes: a second 3D convolution layer, a 2D time adaptive convolution layer, and a third 3D convolution layer connected in sequence; The structure of the 2D time adaptive convolution layer includes: a spatial global average pooling layer, a temporal global average pooling layer, a fully connected layer, a first 1D convolution layer, a batch normalization layer, a ReLU layer and a second 1D convolution layer connected in sequence. The sixth residual connection is established between the input end of the temporal global average pooling layer and the output end of the fully connected layer. The output of the second 1D convolution layer is used as the calibration weight of the 2D time adaptive convolution layer. The calibration weight is added by 1 and then multiplied by the basic weight of the 2D time adaptive convolution layer to obtain the convolution weight of each frame of the 2D time adaptive convolution layer.

7. The method for detecting suspicious persons based on video analysis according to claim 1, characterized in that: The contents of the multi-level suspicious behavior table include: suspicious levels, suspicious behaviors corresponding to each suspicious level, and suspicious items corresponding to each suspicious behavior, and a countermeasure description is added to each suspicious level for reference by security personnel.

8. A suspicious person detection device based on video analysis, characterized in that: include: Target detection unit, target tracking unit, behavior recognition unit and matching alarm unit; The target detection unit is used to perform target detection on the video frame through a target detection algorithm to obtain a person target detection result and a suspicious object target detection result; The target tracking unit is used to associate the previous and next frames of the personnel target detection results through a tracking algorithm to obtain a personnel tracking result; The behavior recognition unit is used to identify the personnel tracking results through the behavior recognition algorithm based on the large pre-trained model CLIP and transfer learning to obtain the personnel behavior recognition results, including: inputting the personnel tracking results into the image encoder of the large pre-trained model CLIP after passing through the 2D convolution layer to extract features, inputting the personnel tracking results into the first 3D convolution layer and fusing them with the features extracted by the image encoder, inputting the fused features into the side network module to further extract features, and then passing through the average pooling layer, the linear layer and the Softmax layer in sequence to finally obtain the personnel behavior recognition results; The matching alarm unit is used to match the personnel behavior recognition results and the suspicious object target detection results with a preset multi-level suspicious behavior table, evaluate the suspicious level of the behavior according to the matching results and send out a corresponding alarm signal.

9. A suspicious person detection device based on video analysis, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the suspicious person detection method based on video analysis as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the suspicious person detection method based on video analysis described in any one of claims 1 to 7 is implemented.

Images