Anomaly behavior detection method, system, device, medium and program product
By introducing a bidirectional long-term motion attention module and a temporal displacement module into the 2D CNN architecture, the problems of high computational cost and poor timeliness in existing video behavior detection are solved, achieving lightweight real-time abnormal behavior detection and improving detection accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2024-03-14
- Publication Date
- 2026-07-24
AI Technical Summary
Existing deep learning methods are computationally expensive and have poor timeliness in video behavior detection, making it difficult to meet real-time monitoring requirements. Furthermore, existing models have limited ability to extract motion information over time in videos, resulting in limited detection accuracy.
In a 2D CNN architecture, a bidirectional long-term motion attention module and a temporal shift module are introduced. Attention weights are obtained by calculating the difference between forward and backward features, and the motion feature map is processed by combining temporal shift, so as to realize the extraction of long-term motion features and the interaction of temporal features. A lightweight architecture is adopted to reduce computational complexity.
It enables the extraction and real-time detection of long-term motion features without increasing additional computational costs, improving detection accuracy. It is suitable for lightweight deployment and for detecting abnormal behavior in surveillance videos.
Smart Images
Figure CN117994859B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of behavior recognition technology, and in particular to an abnormal behavior detection method, system, device, medium, and program product. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] With the coverage of surveillance cameras, it is possible to detect and identify various behaviors of pedestrians, such as abnormal behavior, in public places. Intelligent video surveillance can use intelligent video analysis technology to detect and identify various behaviors of pedestrians from massive amounts of surveillance video, which helps to reduce the harm and occurrence rate of public safety emergencies.
[0004] With the development of deep learning technology, deep learning methods can be applied to detect abnormal pedestrian behavior. However, most deep learning-based methods focus on improving accuracy, while the computational cost and timeliness of the models need further improvement for real-time monitoring applications.
[0005] For example, some studies use 2D Convolutional Neural Networks (CNNs) to detect behavior in videos. Specifically, they divide the video into segments using sparse sampling, extract video frames from each segment, and then use 2D CNNs to predict behavior in the video. However, 2D CNNs cannot extract temporal motion information from the video, which limits their ability to detect motion and thus affects detection accuracy.
[0006] Another type of network for behavior detection in videos combines a recurrent neural network (RNN) with encoding capabilities with a CNN that can extract spatial features of images, such as the convolutional long short-term memory (ConvLSTM) network. This is a model that combines convolutional neural networks and long short-term memory networks to process image data in time series. However, this method cannot capture and encode long-term temporal information.
[0007] To extract features across the temporal scale of videos, many methods for action recognition using 3D CNNs have been proposed. However, 3D CNN models suffer from large size and high computational cost, making them difficult to deploy in real-time detection applications.
[0008] In addition, there are some models that use dual-stream input for behavior detection, such as Flow Gated Network. However, optical flow calculations in dual-stream input are complex and data processing is cumbersome, making it difficult to meet the requirements of real-time detection.
[0009] There are also methods for human pose estimation by extracting key points, but these methods require filtering out too much information from the image and may introduce additional errors when extracting key points, leading to misjudgments of violent behavior. Summary of the Invention
[0010] To address the aforementioned issues, this invention proposes an abnormal behavior detection method, system, device, medium, and program product. It designs an end-to-end model for real-time detection of abnormal behavior in surveillance videos, and introduces a bidirectional long-term motion attention module and a temporal displacement module into a 2D CNN architecture to achieve the extraction of long-term motion features and the interaction of temporal features without increasing additional computational costs, thus exhibiting lightweight characteristics.
[0011] To achieve the above objectives, the present invention adopts the following technical solution:
[0012] In a first aspect, the present invention provides an abnormal behavior detection method, comprising:
[0013] Obtain a sequence of consecutive frame images from the video stream of the behavior to be processed;
[0014] For the current frame image and its next frame image, calculate the forward feature difference and the backward feature difference between the two. Based on the forward feature difference and the backward feature difference, calculate the forward attention weight and the backward attention weight respectively, thereby obtaining the attention feature map.
[0015] The attention feature map is weighted according to the current frame image, and the weighted attention feature map is concatenated with the current frame image to obtain the motion feature map of the current frame image;
[0016] All motion feature maps corresponding to a continuous frame image sequence are reconstructed into single feature maps. The single feature maps are then shifted along the time dimension, and abnormal behavior is identified based on the single feature maps after the shifting process.
[0017] As an alternative implementation, the continuous frame image sequence is as follows: the video stream of the behavior to be processed is sampled at equal intervals, and the 3T sampled images are converted into grayscale images to form a grayscale image sequence; the grayscale image sequence is grouped into groups of three consecutive grayscale images, and then spliced in the channel dimension to obtain three-channel pseudo-images, thereby converting T three-channel pseudo-images into a continuous frame image sequence.
[0018] As an alternative implementation, the forward feature difference and the backward feature difference are respectively:
[0019]
[0020] Among them, M f (t) is the forward feature difference; M b (t+1) is the backward feature difference; Conv2 is a 3×3 convolution; F r (t) and F r (t+1) represents the feature map of the current frame image and the feature map of its next frame image; T represents the number of images in the continuous frame image sequence.
[0021] As an alternative implementation, before calculating the forward feature difference and the backward feature difference, the feature map of the consecutive frame image sequence is downsampled, and the number of channels is reduced according to the channel attenuation ratio.
[0022] After obtaining the forward and backward feature differences, 1×1 convolutional and connection layers are used to recover both the forward and backward feature differences, and forward and backward attention weights are generated by the sigmoid function respectively.
[0023] The attention feature map is obtained by averaging the forward attention weights and backward attention weights;
[0024] The attention feature map is multiplied by the feature map of the current frame image to weight the attention feature map. The weighted attention feature map is then dropped and connected with the feature map of the current frame image to obtain the motion feature map of the current frame image.
[0025] As an alternative implementation, the process of channel displacement of a single feature map along the time dimension includes: reshaping T three-channel motion feature maps into a single feature map with dimensions [H×W, T, C], where C is the number of channels, and H and W are the width and height, respectively; moving some channels in the single feature map along the time dimension, wherein half of the selected channels are moved backward by one step and the other half are moved forward by one step, and the empty spaces above and below after the movement are filled with zeros; and reshaping the single feature map after channel displacement to restore the T feature maps.
[0026] As an alternative implementation, in the process of identifying abnormal behavior, a parallel main classifier and auxiliary classifier are used. The classification results of the main classifier and auxiliary classifier are averaged and pooled to obtain the total classification result. Each frame in the continuous frame image sequence obtains a total classification result. Based on multiple total classification results, average pooling is used to obtain the abnormal behavior detection result corresponding to the video stream of the behavior to be processed.
[0027] Secondly, the present invention provides an abnormal behavior detection system, comprising:
[0028] The acquisition module is configured to acquire a sequence of consecutive frame images of the video stream of the behavior to be processed;
[0029] The attention module is configured to calculate the forward feature difference and the backward feature difference between the current frame image and its next frame image, and calculate the forward attention weight and the backward attention weight based on the forward feature difference and the backward feature difference, thereby obtaining the attention feature map.
[0030] The motion extraction module is configured to weight the attention feature map based on the current frame image, and then concatenate the weighted attention feature map with the current frame image to obtain the motion feature map of the current frame image.
[0031] The temporal displacement and classification module is configured to reshape all motion feature maps corresponding to a continuous frame image sequence into a single feature map, perform channel displacement on the single feature map along the time dimension, and identify abnormal behavior based on the single feature map after channel displacement processing.
[0032] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.
[0033] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.
[0034] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.
[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0036] This invention proposes an end-to-end model for real-time detection of abnormal behavior in surveillance videos. It introduces a bidirectional long-term motion attention module and a temporal shift module into a 2D CNN architecture. The bidirectional long-term motion attention module can be incorporated into any 2D CNN structure, adding additional dynamic feature extraction capabilities to its existing dynamic feature extraction, thus enabling the extraction of long-term motion features. The temporal shift module fuses features from adjacent time periods into a single feature map to represent motion information, achieving interaction of temporal features. Through the cooperation of these two modules, the feature map of a single time segment can encompass long-term motion information related to behavior detection.
[0037] This invention addresses the high computational complexity of behavior detection in videos by proposing a lightweight architecture. 2D CNNs, as backbone networks, typically require significantly less computation than 3D CNNs. The designed bidirectional long-term motion attention mechanism introduces only minimal computation, and the designed temporal displacement module effectively extends the temporal perception range without adding extra computational cost. Therefore, the model has a lightweight architecture, facilitating real-time detection deployment.
[0038] In the classification stage, this invention improves the model's classification ability by using a main classifier and an auxiliary classifier. In order to ensure that the optimization directions of the main classifier and the auxiliary classifier are consistent, the structure of the auxiliary classifier is similar to that of the main classifier. The two classifiers operate in parallel, and finally, the abnormal behavior detection results corresponding to the video segment are obtained by average pooling.
[0039] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0041] Figure 1 This is a flowchart of the abnormal behavior detection method provided in Embodiment 1 of the present invention;
[0042] Figure 2 This is a diagram of the lightweight abnormal behavior detection framework based on the Efficient-B0 network provided in Embodiment 1 of the present invention.
[0043] Figure 3 This is a schematic diagram of the EfficientNet-B0 network structure;
[0044] Figure 4(a) shows the inverse residual module structure of the EfficietNet-B0 network;
[0045] Figure 4(b) is a structural diagram of the inverted residual module that integrates a bidirectional long-term motion attention module and a temporal displacement module provided in Embodiment 1 of the present invention;
[0046] Figure 5 This is a schematic diagram of the bidirectional long-term motion attention module structure provided in Embodiment 1 of the present invention;
[0047] Figure 6 This is a schematic diagram of the timing shift module provided in Embodiment 1 of the present invention;
[0048] Figure 7 This is a schematic diagram of the auxiliary loss branch structure provided in Embodiment 1 of the present invention. Detailed Implementation
[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0050] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.
[0052] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0053] Example 1
[0054] This embodiment provides a method for detecting abnormal behavior, such as Figures 1-2 As shown, it includes:
[0055] Obtain a sequence of consecutive frame images from the video stream of the behavior to be processed;
[0056] For the current frame image and its next frame image, calculate the forward feature difference and the backward feature difference between the two. Based on the forward feature difference and the backward feature difference, calculate the forward attention weight and the backward attention weight respectively, thereby obtaining the attention feature map.
[0057] The attention feature map is weighted according to the current frame image, and the weighted attention feature map is concatenated with the current frame image to obtain the motion feature map of the current frame image;
[0058] All motion feature maps corresponding to a continuous frame image sequence are reconstructed into single feature maps. The single feature maps are then shifted along the time dimension, and abnormal behavior is identified based on the single feature maps after the shifting process.
[0059] In this embodiment, after obtaining the video stream of the behavior to be processed, the video stream is sampled at uniform intervals to reduce data redundancy. The sampled RGB images are converted into grayscale images to obtain a grayscale image sequence. The grayscale image sequence is grouped into groups of three consecutive grayscale images and spliced in the channel dimension to form a three-channel pseudo-image, thereby forming a pseudo-RGB image sequence.
[0060] Specifically:
[0061] (1) Sample the frames in the video stream at uniform intervals and extract 3T RGB images to form an RGB image sequence F = [F1, F2, ..., F]. 3T ].
[0062] (2) Use the standard weighted average method to convert the sampled RGB image into a single-channel grayscale image to form a grayscale image sequence.
[0063] I(x,y)=W R I R (x,y)+W G I G (x,y)+W B I B (x,y) (1)
[0064] Among them, I R I G I B These represent the pixel intensities of the red, green, and blue channels, respectively. Based on the human eye's brightness perception system, the weights W... R =0.30, W G =0.59, W B =0.11.
[0065] (3) Group the grayscale image sequence into groups of three consecutive grayscale images, and stitch them together along the channel dimension to obtain a three-channel pseudo-image. Then, convert the T three-channel pseudo-images into a pseudo-RGB image sequence (i.e., a continuous frame image sequence) I = [I1, I2, ..., I... T ].
[0066] In the image processing domain, attention mechanisms, much like the human visual system, can give more attention to relevant information. In the field of video-based action recognition, many attention mechanism modules have been introduced into models to enhance spatiotemporal information and further improve model performance. Motion attention mechanisms can model motion information in a set of continuous static images, giving greater weight to channels representing motion, and achieving long-range motion modeling using 2D CNN networks by capturing inter-segment information differences.
[0067] Therefore, in this embodiment, EfficientNet-B0 from a two-dimensional convolutional neural network (2D CNN) is used as the backbone network, such as... Figure 3 As shown, a bidirectional long-term motion attention module and a temporal shift module are integrated to realize long-term motion feature extraction and temporal information interaction, respectively, thereby constructing a feature extraction network. The bidirectional long-term motion attention module and the temporal shift module are integrated into the inverse residual module of the EfficientNet-B0 network. The structure of the inverse residual module is shown in Figure 4(a), and the structure of the inverse residual module with the bidirectional long-term motion attention module and the temporal shift module is shown in Figure 4(b).
[0068] Specifically, based on the temporal difference between pseudo-images of consecutive frames, a lightweight bidirectional long-term motion attention module is designed to represent motion information in a single feature map using the difference features of adjacent time periods, thereby extracting long-term motion features. The temporal displacement module fuses the features of adjacent time periods into a single feature map to represent motion information, thereby realizing the interaction of temporal features. Through the cooperation of the two modules, the feature map of a single time segment can cover long-term motion information related to behavior detection.
[0069] In this embodiment, a pseudo-RGB image sequence is input into a feature extraction network for feature extraction, realizing spatial feature extraction, motion feature extraction, and temporal information interaction, and outputting features with the ability to distinguish abnormal behavior.
[0070] like Figure 5 As shown, the processing flow of the bidirectional long-term motion attention module includes the following steps:
[0071] A. After the pseudo-RGB image sequence is input into the 2D CNN network, a feature map F is first extracted from each frame of the image. Then, in order to meet the requirements of lightweight design, a 1×1 2D CNN (Conv1) is first used to downsample the feature map F.
[0072]
[0073] Among them, F r represents the downsampled feature map; r represents the channel attenuation ratio, which is determined by the number of channels in the 2D CNN network. Since the number of image channels has been reduced by the first 1×1 convolution in the inverse residual module, r is used as the channel attenuation ratio here; * represents the convolution operation; T is the number of pseudo-RGB images, which also represents the time segment; C represents the number of channels; H and W represent the width and height of the feature map, respectively.
[0074] B. For the feature map corresponding to the pseudo-RGB image of the current frame, calculate the forward feature difference and backward feature difference between the feature maps of two adjacent frames to obtain motion information.
[0075] Specifically, a 3×3 2D CNN (Conv2) is used as the mapping function to map the feature maps F of two adjacent frames. r (t+1) and F r (t), calculate bidirectional motion information to address the spatial misalignment problem:
[0076]
[0077] Among them, M f (t) is the forward feature difference, M b (t+1) is the backward feature difference.
[0078] For the last time step t, the bidirectional motion information is set to M. f (T) = 0 and M b (T) = 0.
[0079] C. For the forward and backward feature differences, a 1×1 2D CNN (Conv3) and a concatenation layer (Concat) are used to recover the original shape of the input feature map F. Then, the forward attention weights W are generated using the sigmoid function. f and backward attention weights W b :
[0080]
[0081] D. Based on the forward attention weights W f and backward attention weights W b Calculate attention feature map A:
[0082] A = 0.5W f +0.5W b (5)
[0083] E. Using the Hadamard product, drop-connect operation, and residual connection method, obtain the motion feature map F′ of the current frame image:
[0084]
[0085] Where ⊙ represents the Hadamard product operation to obtain weighted features; The + indicates a drop-connect operation, which randomly sets the weights to zero with a certain probability to improve the generalization ability of the neural network model; the + indicates a residual connection operation, which fuses the original features with the weighted features of the attention feature map, preserving the original features while utilizing a bidirectional long-term motion attention mechanism.
[0086] In this embodiment, the improved EfficientNet-B0 network also employs a bidirectional temporal shift module to further enhance the network's ability to model time and achieve interaction of spatiotemporal features. For example... Figure 6 As shown, firstly, T 3-channel motion feature maps (if there are fewer than T motion feature maps, the remaining portion is padded with zeros) are reshaped into single feature maps of dimension [H×W, T, C]. Then, a portion (e.g., 1 / 8, 1 / 16, or 1 / 32) of the single feature map is shifted along the time dimension. Half of the selected channels is shifted backward by one step, and the other half is shifted forward by one step, with the empty spaces above and below filled with zeros. Finally, the resulting tensor is reshaped to restore the T feature maps, with the same format as the input. Clearly, the features at any time step t are fused with the features at time steps t-1 and t+1. By using the temporal shift module, the temporal domain sensing range is effectively expanded without increasing computational cost.
[0087] In this embodiment, based on the output features of the feature extraction network, a softmax classifier is used to achieve classification. Each time segment corresponding to the pseudo-RGB image yields a classification result. Based on multiple classification results, average pooling is used to obtain the abnormal behavior detection result corresponding to the video segment.
[0088] like Figure 2 As shown, this embodiment improves the model's classification ability through a main classifier and an auxiliary classifier. Both the main and auxiliary classifiers use the Softmax classifier. In fact, the auxiliary classifier is only used during the model training phase and is discarded during the inference phase. This embodiment inserts the auxiliary classifier after the 7th phase of the improved EfficientNet-B0, as shown... Figure 7 As shown.
[0089] To ensure consistency in the optimization directions of the main classifier and the auxiliary classifier, the auxiliary classifier structure is similar to that of the main classifier. The two classifiers operate in parallel, each providing an instantaneous score via Softmax. Based on these scores, the instantaneous cross-entropy loss in each branch can be calculated. The main loss L is obtained by average pooling the instantaneous cross-entropy loss. main and L aux , by L total =L main +λL aux The total loss was calculated.
[0090] To evaluate the performance of the model proposed in this embodiment, evaluations were conducted on the RWF-2000 and Hockey Fight datasets. The RWF-2000 dataset contains anomalous and non-nominal behaviors from surveillance videos in different scenarios. It consists of video clips from 2000 surveillance cameras captured in real-world scenes, with each video containing 150 frames. The dataset has been divided into training and testing sets, and therefore, the ablation experiments primarily used the results from the RWF-2000 dataset for evaluation. The Hockey Fight dataset contains 1000 action clips collected from ice hockey games, with 500 videos showing normal actions and 500 videos showing anomalous actions. Each clip consists of approximately 40 frames of 360×288 pixels. Each clip was manually labeled, but the Hockey Fight dataset represents behavior in a single scenario, which has certain limitations.
[0091] The comparative experimental results are shown in Table 1. As can be seen from the experimental structure, the method in this embodiment achieved an accuracy of 98.5% on the Hockey Fight dataset and 90.25% on the RWF-2000 dataset. Simultaneously, the computational complexity is 1.21 G FLOPs (FLOPs refers to the number of floating-point operations, a unit describing the total computational cost), and the number of parameters is 4.48 M. This demonstrates a lightweight 2D CNN architecture that achieves good performance in detecting abnormal behavior in surveillance videos.
[0092] Table 1 Experimental Results
[0093] 98.5％ 90.25％ 1.21 4.48
[0094] Example 2
[0095] This embodiment provides an abnormal behavior detection system, including:
[0096] The acquisition module is configured to acquire a sequence of consecutive frame images of the video stream of the behavior to be processed;
[0097] The attention module is configured to calculate the forward feature difference and the backward feature difference between the current frame image and its next frame image, and calculate the forward attention weight and the backward attention weight based on the forward feature difference and the backward feature difference, thereby obtaining the attention feature map.
[0098] The motion extraction module is configured to weight the attention feature map based on the current frame image, and then concatenate the weighted attention feature map with the current frame image to obtain the motion feature map of the current frame image.
[0099] The temporal displacement and classification module is configured to reshape all motion feature maps corresponding to a continuous frame image sequence into a single feature map, perform channel displacement on the single feature map along the time dimension, and identify abnormal behavior based on the single feature map after channel displacement processing.
[0100] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.
[0101] In further embodiments, the following is also provided:
[0102] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.
[0103] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0104] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.
[0105] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.
[0106] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.
[0107] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.
[0108] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.
[0109] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.
[0110] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and so on. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.
[0111] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0112] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for detecting abnormal behavior, characterized in that, include: Obtain a sequence of consecutive frame images from the video stream of the behavior to be processed; For the current frame image and its next frame image, calculate the forward feature difference and the backward feature difference between the two. Based on the forward feature difference and the backward feature difference, calculate the forward attention weight and the backward attention weight respectively, thereby obtaining the attention feature map. Wherein, the forward feature difference and the backward feature difference are respectively: ； In the formula, It is the forward feature difference; It is the backward feature difference; Conv2 is a 3×3 convolution; and The feature map of the current frame image and the feature map of its next frame image; T is the number of images in the consecutive frame image sequence; The attention feature map is weighted according to the current frame image, and the weighted attention feature map is concatenated with the current frame image to obtain the motion feature map of the current frame image; Before calculating the forward and backward feature differences, the feature maps of the consecutive frame image sequence are downsampled, and the number of channels is reduced according to the channel attenuation ratio. After obtaining the forward and backward feature differences, 1×1 convolutional and connection layers are used to recover both the forward and backward feature differences, and forward and backward attention weights are generated by the sigmoid function respectively. The attention feature map is obtained by averaging the forward attention weights and backward attention weights; The attention feature map is multiplied by the feature map of the current frame image to weight the attention feature map. The weighted attention feature map is then dropped and connected with the feature map of the current frame image to obtain the motion feature map of the current frame image. All motion feature maps corresponding to a continuous frame image sequence are reconstructed into single feature maps. The single feature maps are then shifted along the time dimension, and abnormal behavior is identified based on the single feature maps after the channel shifting process. The process of channel displacement of a single feature map along the time dimension includes: reshaping T three-channel motion feature maps into a single feature map with dimensions [H×W, T, C], where C is the number of channels, and H and W are the width and height, respectively; moving some channels in the single feature map along the time dimension, wherein half of the selected channels are moved backward by one step and the other half are moved forward by one step, and the empty spaces above and below are filled with zeros; and reshaping the single feature map after channel displacement to restore the T feature maps.
2. The abnormal behavior detection method as described in claim 1, characterized in that, The continuous frame image sequence is as follows: the video stream of the behavior to be processed is sampled at equal intervals, and the 3T sampled images are converted into grayscale images to form a grayscale image sequence; the grayscale image sequence is grouped into groups of three consecutive grayscale images, and then stitched together along the channel dimension to obtain three-channel pseudo-images, thereby converting T three-channel pseudo-images into a continuous frame image sequence.
3. The abnormal behavior detection method as described in claim 1, characterized in that, In the process of identifying abnormal behavior, a parallel main classifier and auxiliary classifier are used. The classification results of the main classifier and auxiliary classifier are averaged and pooled to obtain the overall classification result. Each frame in the continuous frame image sequence obtains an overall classification result. Based on multiple overall classification results, average pooling is used to obtain the abnormal behavior detection result corresponding to the video stream of the behavior to be processed.
4. An abnormal behavior detection system, characterized in that, include: The acquisition module is configured to acquire a sequence of consecutive frame images of the video stream of the behavior to be processed; The attention module is configured to calculate the forward feature difference and the backward feature difference between the current frame image and its next frame image, and calculate the forward attention weight and the backward attention weight based on the forward feature difference and the backward feature difference, thereby obtaining the attention feature map. Wherein, the forward feature difference and the backward feature difference are respectively: ； In the formula, It is the forward feature difference; It is the backward feature difference; Conv2 is a 3×3 convolution; and The feature map of the current frame image and the feature map of its next frame image; T is the number of images in the consecutive frame image sequence; The motion extraction module is configured to weight the attention feature map based on the current frame image, and then concatenate the weighted attention feature map with the current frame image to obtain the motion feature map of the current frame image. Before calculating the forward and backward feature differences, the feature maps of the consecutive frame image sequence are downsampled, and the number of channels is reduced according to the channel attenuation ratio. After obtaining the forward and backward feature differences, 1×1 convolutional and connection layers are used to recover both the forward and backward feature differences, and forward and backward attention weights are generated by the sigmoid function respectively. The attention feature map is obtained by averaging the forward attention weights and backward attention weights; The attention feature map is multiplied by the feature map of the current frame image to weight the attention feature map. The weighted attention feature map is then dropped and connected with the feature map of the current frame image to obtain the motion feature map of the current frame image. The temporal displacement and classification module is configured to reshape all motion feature maps corresponding to a continuous frame image sequence into a single feature map, perform channel displacement on the single feature map along the time dimension, and identify abnormal behavior based on the single feature map after channel displacement processing. The process of channel displacement of a single feature map along the time dimension includes: reshaping T three-channel motion feature maps into a single feature map with dimensions [H×W, T, C], where C is the number of channels, and H and W are the width and height, respectively; moving some channels in the single feature map along the time dimension, wherein half of the selected channels are moved backward by one step and the other half are moved forward by one step, and the empty spaces above and below are filled with zeros; and reshaping the single feature map after channel displacement to restore the T feature maps.
5. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-3.
6. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-3.
7. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-3.