Abnormal behavior recognition method and device and storage medium

Abstract

The invention discloses an abnormal behavior recognition method and device and a storage medium, and relates to the technical field of video analysis, and the method comprises the steps: obtaining an original video sequence, and carrying out the frame extraction, size adjustment and normalization of the original video sequence, and obtaining a video frame sequence; calculating an inter-frame difference image between adjacent video frames in the video frame sequence to obtain a video frame difference sequence; performing feature extraction on the video frame sequence and the video frame difference sequence to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence; performing abnormal region feature enhancement on the frame feature and the frame difference feature to generate a first attention feature map and a second attention feature map; performing feature fusion on the first attention feature map and the second attention feature map to generate fusion features; and carrying out abnormal behavior discrimination on the fusion features to obtain an abnormal behavior identification result. The abnormal behavior recognition result can be quickly and accurately given, and powerful technical support is provided for the fields of video monitoring and safety protection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video analysis, in particular to an abnormal behavior recognition method and device and a storage medium. BACKGROUND

[0002] With the wide deployment of intelligent monitoring systems, video abnormal behavior recognition technology has become an important research direction of public safety and intelligent monitoring. Existing abnormal recognition methods mainly fall into the following categories: a. Methods based on manual features: Early video anomaly detection relies mainly on optical flow features, trajectory features or motion histograms and other manual features. This kind of method is very sensitive to scene changes, light differences and occlusions, and is difficult to adapt to complex environments. b. Methods based on convolutional neural networks: With the development of deep learning, researchers began to use CNN to extract spatial features of video frames, and realized anomaly detection through fully connected layers or autoencoders. However, CNN models mainly capture static image information and lack the ability to perceive temporal dynamic changes. SUMMARY

[0003] The main purpose of the present application is to provide an abnormal behavior recognition method, device and storage medium, aiming at solving the technical problems of poor adaptability to complex environments and lack of temporal dynamic change perception ability of the existing abnormal behavior recognition method.

[0004] To achieve the above purpose, the present application provides an abnormal behavior recognition method, which comprises: obtaining an original video sequence, and performing frame extraction, size adjustment and normalization on the original video sequence to obtain a video frame sequence; calculating the inter-frame difference image between adjacent video frames in the video frame sequence to obtain a video frame difference sequence; respectively extracting features from the video frame sequence and the video frame difference sequence to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence; respectively enhancing abnormal region features of the frame features and the frame difference features to generate a first attention feature map and a second attention feature map; fusing the first attention feature map and the second attention feature map to generate a fusion feature; discriminating abnormal behaviors from the fusion feature to obtain an abnormal behavior recognition result.

[0005] In an embodiment, the step of respectively extracting features from the video frame sequence and the video frame difference sequence to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence comprises: inputting the video frame sequence and the video frame difference sequence into a temporal distribution convolution network respectively to extract spatial features, to obtain video frame spatial features and video frame difference spatial features; inputting the video frame spatial features and the video frame difference spatial features into a separated convolution long short-term memory network respectively to extract temporal features, to obtain video frame temporal features and video frame difference temporal features; obtaining frame features of the video frame sequence according to the video frame temporal features, and obtaining frame difference features of the video frame difference sequence according to the video frame difference temporal features.

[0006] In an embodiment, the step of performing feature fusion on the first attention feature map and the second attention feature map to generate a fusion feature comprises: performing spatial feature compression on the first attention feature map and the second attention feature map respectively to obtain first channel-level statistical features and second channel-level statistical features; concatenating the first channel-level statistical features and the second channel-level statistical features into a feature vector to obtain a concatenated statistical feature; performing feature fusion on the concatenated statistical feature to generate a fusion feature.

[0007] In an embodiment, the step of performing feature fusion on the concatenated statistical feature to generate a fusion feature comprises: inputting the concatenated statistical feature into a multi-layer perceptron to generate a normalized weight, the normalized weight comprising a first weight and a second weight; performing feature fusion on the first weight, the second weight, the first attention feature map and the second attention feature map to generate a fusion feature.

[0008] In an embodiment, the step of performing abnormal region feature enhancement on the frame features and the frame difference features respectively to generate a first attention feature map and a second attention feature map comprises: inputting the frame features and the frame difference features into a channel compression convolution layer respectively to perform convolution, to obtain a first spatial feature map and a second spatial feature map; inputting the first spatial feature map and the second spatial feature map into an activation layer respectively to perform activation, to obtain a first activation feature map and a second activation feature map; performing abnormal region feature enhancement on the first activation feature map and the second activation feature map respectively to obtain a first attention feature map and a second attention feature map.

[0009] In an embodiment, the step of performing abnormal behavior discrimination on the fusion feature to obtain an abnormal behavior recognition result comprises: performing dimension compression on the fusion feature to obtain a global feature; The global features are subjected to dimensionality reduction and nonlinear transformation to obtain hidden features; The hidden features are mapped to abnormal behaviors to obtain the abnormal probability of the abnormal behavior; The abnormal behavior identification result is obtained based on the abnormal probability.

[0010] In one embodiment, the step of obtaining the abnormal behavior identification result based on the abnormal probability includes: Obtain the preset probability threshold; Compare the anomaly probability with the preset probability threshold; When the abnormal probability is greater than or equal to the preset probability threshold, the abnormal behavior result is determined to be that abnormal behavior exists.

[0011] In one embodiment, the step of obtaining the original video sequence and performing frame extraction, resizing, and normalization on the original video sequence to obtain a video frame sequence includes: Obtain the original video sequence, preset sampling strategy, and target size; The original video sequence is frame-by-frame extracted using the preset strategy to obtain extracted video frames. The extracted video frames are subjected to channel conversion to obtain converted video frames; The converted video frame is adjusted to the target size to obtain a resized video frame; The resized video frames are normalized to obtain a video frame sequence.

[0012] Furthermore, to achieve the above objectives, this application also proposes an abnormal behavior identification device, which includes: The acquisition module is used to acquire the original video sequence and perform frame extraction, size adjustment and normalization on the original video sequence to obtain a video frame sequence; The calculation module is used to calculate the inter-frame difference image between adjacent video frames in the video frame sequence to obtain the video frame difference sequence; The feature extraction module is used to extract features from the video frame sequence and the video frame difference sequence respectively, to obtain the frame features of the video frame sequence and the frame difference features of the video frame difference sequence. The feature enhancement module is used to enhance the abnormal region features of the frame features and the frame difference features respectively, and generate a first attention feature map and a second attention feature map; The feature fusion module is used to fuse the first attention feature map and the second attention feature map to generate fused features; The discrimination module is used to discriminate abnormal behavior based on the fused features and obtain abnormal behavior identification results.

[0013] In addition, to achieve the above objectives, this application also proposes an abnormal behavior recognition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the abnormal behavior recognition method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the abnormal behavior recognition method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the abnormal behavior recognition method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: 1) The abnormal behavior recognition method proposed in this application effectively improves the accuracy and robustness of abnormal behavior detection by integrating dual information from video frame sequences and inter-frame difference images. This method not only captures static features in the video but also captures dynamic change information through inter-frame difference images; the combination of these two aspects makes the recognition of abnormal behavior more comprehensive. Furthermore, by introducing an attention mechanism to enhance features in key regions, the model's sensitivity to abnormal behavior is further improved, reducing false positives and false negatives. Finally, through feature fusion and abnormal behavior discrimination, it can quickly and accurately provide abnormal behavior recognition results, offering strong technical support for video surveillance, security protection, and other fields.

[0017] 2) Extracting spatial features through temporally distributed convolutional networks can effectively extract detailed information and important features from video frames, thereby improving the model's ability to understand video content. Spatial features can capture visual information about static objects, backgrounds, and important dynamic areas. Using separable convolutional long short-term memory networks for temporal feature extraction can effectively identify patterns of dynamic changes in videos. Long short-term memory networks excel at processing time-series data, capturing temporal dependencies between video frames, and identifying temporal behaviors such as motion trajectories and event sequences, which is particularly important for the analysis of moving objects. Feature extraction through separable convolution can reduce model complexity, making it easier to generalize and less prone to overfitting. This is especially important when processing video data, as video data often contains a lot of redundant information. Combining temporally distributed convolutional networks and separable convolutional long short-term memory networks can optimize the utilization of computational resources, improve frame processing speed, and enable the system to perform real-time analysis and response more efficiently. By extracting features from video frames and frame differences separately, the events and changes occurring in the video can be understood more clearly, thereby improving the interpretability of the model. Frame difference features can highlight motion changes, thereby helping to identify action or state changes.

[0018] 3) By spatially compressing the attention feature map, the most relevant statistical features for the task can be effectively extracted. This selective aggregation of information enhances the model's focus on key features, thereby strengthening its performance on specific tasks. Concatenating feature vectors not only preserves important information from the first and second attention feature maps but also increases the feature dimensionality. This concatenation operation allows the model to learn diverse information from different feature maps, contributing to a more comprehensive feature representation. By fusing the statistical information of different attention features, the model can better capture complex patterns and relationships, especially when dealing with tasks with multiple important information sources. This capability makes the model more flexible in recognizing and parsing multiple features. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the abnormal behavior identification method of this application. Figure 2 This is a schematic diagram of the structure of an abnormal attention module provided in an embodiment of the abnormal behavior recognition method of this application; Figure 3 A schematic diagram of the system structure provided for an embodiment of the abnormal behavior recognition method of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the abnormal behavior identification method of this application; Figure 5 This is a flowchart illustrating Embodiment 3 of the abnormal behavior identification method of this application; Figure 6 A schematic diagram of the adaptive weighted fusion module provided in an embodiment of the abnormal behavior recognition method of this application; Figure 7 A simplified flowchart is provided for one embodiment of the abnormal behavior identification method of this application; Figure 8 This is a schematic diagram of the module structure of the abnormal behavior recognition device according to an embodiment of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the abnormal behavior recognition method in this application embodiment.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] To capture temporal features, some existing technologies combine convolutional neural networks with long short-term memory networks, with video frame sequences as input. This method improves the recognition effect to some extent, but has the following problems: (1) CNN and LSTM modules extract features independently, and the fusion of spatial and temporal features is insufficient; (2) The model has a large number of parameters and is complex to train, and is prone to overfitting to small sample abnormal data; (3) In complex scenes, the model has difficulty in accurately distinguishing between normal motion and abnormal behavior. Some other studies use frame sequences and inter-frame difference images as dual input branches, and use convolutional networks to extract static and dynamic features before fusing the output. This structure enhances motion sensitivity to some extent, but still has the following defects: the dual-branch fusion method is fixed, such as simple splicing or averaging, and it cannot adaptively adjust the weights according to the dynamic changes of the video; the ability to focus on abnormal targets is insufficient, and the model has difficulty locating abnormal areas; the feature expression of abnormal behavior is easily interfered with by background noise, resulting in unstable detection results. In summary, existing video anomaly detection technologies suffer from the following shortcomings: a. Static feature fusion methods: Existing dual-stream networks lack an adaptive mechanism in the feature fusion stage, failing to dynamically adjust the contribution of "frame features" and "frame difference features" based on scene motion complexity, resulting in significant performance fluctuations across different scenarios. b. Lack of attention mechanisms for anomaly regions: Traditional models fail to explicitly model the importance of anomaly regions after extracting spatiotemporal features, leading to a scattered focus and insufficient response to real anomalies. c. Weak robustness to complex dynamic environments: When changes in lighting, background motion, or camera shake occur in the video, existing methods easily misclassify normal changes as anomalies. d. Insufficient model interpretability: Current anomaly detection models often only output anomaly probabilities, lacking interpretable descriptions of the anomaly source or region, making reliable applications in fields such as security monitoring difficult.

[0026] This application provides a solution, proposing a video abnormal behavior recognition method based on adaptive frame difference dynamic weighting and anomaly attention enhancement mechanism. An adaptive weight fusion module is introduced between the two-modal branches to realize dynamic weight adjustment of different modal features. An anomaly attention module is added in the spatiotemporal feature extraction stage to automatically highlight the feature response of abnormal regions. Thus, while ensuring the model is lightweight, the accuracy, stability and interpretability of anomaly detection are significantly improved.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or abnormal behavior recognition device capable of performing the above functions. The following description uses an abnormal behavior recognition device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a method for identifying abnormal behavior, referring to...Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the abnormal behavior identification method of this application.

[0029] In this embodiment, the abnormal behavior identification method includes steps S10 to S60: Step S10: Obtain the original video sequence, and perform frame extraction, size adjustment and normalization on the original video sequence to obtain a video frame sequence.

[0030] It should be noted that the main application scenarios of this embodiment can be: abnormal behavior recognition in the fields of public safety and urban monitoring, abnormal behavior recognition in the fields of traffic management and road safety, abnormal behavior recognition in the fields of industrial and production safety, abnormal behavior recognition in the fields of smart healthcare and elderly care, abnormal behavior recognition in commercial and retail scenarios, and abnormal behavior recognition in the fields of rail transit and airport security. This application can be embedded in urban public safety monitoring systems to achieve automatic identification and early warning of abnormal behaviors in public places such as subways, shopping malls, squares, campuses, and hospitals, such as fighting, running, falling, and abandoned items. It can also be integrated into front-end smart cameras or NVR (Network Video Recorder) devices to achieve real-time anomaly detection and local alarm through edge computing, reducing the load on the back-end server. It can also be used to detect traffic violations such as driving against traffic, sudden stops, U-turns, crossing lines, and pedestrians running red lights. Deploying the abnormal behavior recognition algorithm of this application in transportation infrastructure can enable early detection and alarms for abnormal events such as fires, traffic accidents, and obstructions. It can also be used in industrial manufacturing scenarios to automatically identify operator behavior and the operating status of production equipment, promptly alarming when personnel misoperation, equipment stagnation, or abnormal movements are detected. In high-risk industries such as power, chemical, and mining, this application can monitor abnormal behaviors such as personnel falls, boundary crossings, and sparks, improving operational safety. It can also be used in areas such as hospital wards and rehabilitation centers to automatically identify abnormal behaviors such as patient falls and getting out of bed, assisting medical staff in timely intervention. It can be embedded in smart monitoring cameras or IoT terminals to achieve automatic detection and alarms for abnormal behaviors of the elderly, such as falls and prolonged periods of stillness. In shopping mall / supermarket loss prevention systems, abnormal lingering, boundary crossing, crowding, and shelf damage can be detected for loss prevention and safety analysis. By identifying abnormal customer behavior, such as running or falling, a safe shopping environment can be ensured. For subway car and platform security detection, abnormal behaviors such as pushing, falling, and entering restricted areas can be identified in real time, enabling automatic early warning. For airport and train station security assistance, it can be combined with video security inspection systems to detect suspicious behaviors such as abandoned luggage, abnormal running, or gathering. In the education and campus security field, it can be deployed in school buildings, dormitories, playgrounds, canteens, and other areas to identify abnormal behaviors such as horseplay, falling, and crowding, improving the level of intelligence in campus security monitoring.

[0031] This application can be packaged as an algorithm library or API interface and integrated into monitoring management platforms, video analysis systems, or third-party applications. It can also be deployed as a lightweight model on embedded devices to achieve local real-time anomaly recognition, or as a cloud-based AI service module to interface with video stream access systems, enabling centralized anomaly detection across multiple scenarios and terminals. Furthermore, it can be implemented via mobile apps or mini-programs for remote real-time monitoring and anomaly alarm push notifications.

[0032] It should be noted that the original video sequence can come from various sources, including but not limited to local storage devices, webcams, and video streaming servers. After obtaining the original video sequence, the first step is frame extraction, which involves extracting key video frames from the original video sequence at certain time intervals or frame rates. This step aims to reduce the amount of data processed subsequently while preserving key information in the video. The extracted video frames may have different sizes and formats; to facilitate subsequent processing, these frames need to be resized to a uniform target size. Size resizing can be achieved using image scaling algorithms to ensure spatial consistency across all video frames. The resized video frames then undergo normalization. Normalization is a common data preprocessing method that aims to convert data of different dimensions to the same dimension, facilitating subsequent feature extraction and model training. Normalization can be performed in various ways, such as scaling pixel values ​​to between 0 and 1, or using mean-variance normalization to eliminate differences between different video frames caused by factors such as lighting and contrast.

[0033] In one feasible implementation, step S10 may include steps A11 to A15: Step A11: Obtain the original video sequence, preset sampling strategy, and target size; It should be noted that it can acquire local video files, live network streams, and raw video sequences from USB camera input. Preset sampling strategies can include sampling frequency and sampling interval, for example, setting the sampling frequency to a fixed 15fps, while the sampling interval can be set as needed to retain frames with timestamp differences greater than or equal to the interval, ensuring temporal consistency. The target size can be configured as needed, for example, set to 256×256 pixels.

[0034] Step A12: Extract frames from the original video sequence using the preset strategy to obtain extracted video frames; In practice, the original video sequence can be read by repeatedly calling the reading tool and recording the timestamp of each frame. Frames can then be filtered according to the sampling interval to obtain the extracted video frames.

[0035] Step A13: Perform channel conversion on the extracted video frames to obtain converted video frames; It should be noted that the extracted video frames can be converted to the RGB format commonly used in models. The specific conversion depends on the number of channels in the original video frame. If the original video frame is a grayscale image (i.e., single-channel), it should be copied three times and converted to a three-channel RGB format. If the original video frame is already in RGB format (i.e., three-channel), no conversion is required.

[0036] Step A14: Adjust the converted video frame to the target size to obtain a resized video frame; After obtaining the converted video frames, image scaling algorithms, such as bilinear interpolation and bicubic interpolation, are used to adjust the size of the converted video frames to a preset target size, such as 256×256 pixels, to ensure that all video frames have spatial consistency. Step A15: Normalize the resized video frames to obtain a video frame sequence.

[0037] In practice, the min-max normalization method can be used to scale the pixel values ​​of the resized video frames to between 0 and 1, or the mean-variance normalization method can be used to eliminate differences between different video frames caused by factors such as lighting and contrast, ultimately resulting in a video frame sequence. This sequence will serve as input for subsequent steps, providing a standardized data foundation for feature extraction and model training. This preprocessing ensures the consistency and comparability of the video frame data input into the model, thereby improving the model's recognition accuracy and stability.

[0038] Step S20: Calculate the inter-frame difference image between adjacent video frames in the video frame sequence to obtain the video frame difference sequence.

[0039] It is understandable that the inter-frame difference image between adjacent video frames in a video frame sequence can be calculated. The inter-frame difference image can reflect the changes between adjacent frames and highlight the moving areas. Specifically, for the i-th frame and the (i+1)-th frame in the video frame sequence, the inter-frame difference image is obtained by subtracting the pixel values. Assuming that the pixel value of the i-th frame is F(i) and the pixel value of the (i+1)-th frame is F(i+1), then the formula for calculating the inter-frame difference image D(i) is: D(i) = |F(i+1) - F(i)|.

[0040] By performing this operation on all adjacent frames in a video frame sequence, a video frame difference sequence can be obtained. This sequence provides crucial information for subsequent analysis of motion changes in the video, helping to more accurately identify abnormal behavior. For example, in some surveillance scenarios, the inter-frame difference images of a person walking normally show relatively regular changes, while the inter-frame difference images corresponding to abnormal running or fighting behaviors show more drastic and irregular changes. Analyzing the video frame difference sequence can help initially screen out segments that may contain abnormal behavior.

[0041] Step S30: Extract features from the video frame sequence and the video frame difference sequence respectively to obtain the frame features of the video frame sequence and the frame difference features of the video frame difference sequence.

[0042] It is understandable that feature extraction can be achieved through a bimodal feature extraction module, which includes two independent spatiotemporal feature branches, thereby enabling feature extraction from video frame sequences and video frame difference sequences. This allows for the extraction of spatial and temporal features from the video frame sequences and video frame difference sequences, resulting in frame features of the video frame sequences and frame difference features of the video frame difference sequences.

[0043] Step S40: Perform abnormal region feature enhancement on the frame features and the frame difference features respectively to generate a first attention feature map and a second attention feature map.

[0044] Understandably, anomaly region feature enhancement can be achieved through an anomaly attention module, which automatically focuses on areas in the video where abnormal behavior may exist. For frame features, the anomaly attention module analyzes the spatial features of each region in the video frame, identifying areas that significantly differ from normal behavior patterns, such as suddenly appearing objects or abnormal changes in a person's posture. It then enhances the features of these regions, generating a first attention feature map. This first attention feature map highlights key areas in the video frame where abnormal behavior may exist, allowing subsequent processing to focus more on these areas and improving the accuracy of anomaly behavior recognition.

[0045] Similarly, for frame difference features, the anomaly attention module analyzes the temporal changes in various regions of the inter-frame difference image. Inter-frame difference images of normal behavior typically exhibit relatively regular change patterns, while those of abnormal behavior show drastic and irregular changes. The anomaly attention module can capture these anomalous temporal changes and enhance the corresponding regional features, generating a second attention feature map. This second attention feature map highlights key regions of abnormal motion in the video frame difference sequence, helping to more accurately identify anomalous behavior.

[0046] By using the abnormal region feature enhancement step, the method in this application can further highlight possible abnormal behaviors in the video based on feature extraction.

[0047] In one feasible implementation, step S40 may include steps A21 to A23: Step A21: Convolve the input channels of the frame features and the frame difference features into the compressed convolutional layer to obtain the first spatial feature map and the second spatial feature map; It should be noted that, as Figure 2 As shown, Figure 2This is a schematic diagram of the anomaly attention module. The module includes convolutional layers, attention map generation layers, and attention weighting layers. The anomaly attention module focuses on the features of anomalous regions and suppresses background interference. The input is the frame features or frame difference features output by the bimodal feature extraction module, processed independently in each branch to generate the corresponding attention map. The convolutional layer performs 1×1 convolution on the input features to reduce the dimensionality, decreasing the number of channels from 256 to 1, extracting the spatial response of the anomalous region. This response is then activated by a sigmoid function to generate the attention map, with pixel values ​​ranging from 0 to 1, where 1 represents the anomalous region and 0 represents the background, thus achieving spatial localization of the anomalous region. The attention map is then multiplied element-wise with the original input features to enhance the features of the anomalous region, such as stationary anomalous objects or fast-moving pedestrians, outputting an enhanced feature.

[0048] Therefore, the frame features and frame difference features can be input into the channel compression convolutional layer in the anomaly attention module, respectively. The channel compression convolutional layer then processes the frame features... Perform a 1×1 convolution to compress 256 channels into 1 channel, extracting pixel importance in the spatial dimension. Parameter settings can be: number of kernels: 1 (outputting single-channel spatial features); kernel size: 1×1 (maintaining the spatial size); stride: 1; padding: 0; weight initialization: He normal initialization; bias: 0. Then, use the channel compression convolution layer to extract frame difference features. A 1×1 convolution is performed to compress 256 channels into 1 channel. The pixel importance parameters for the spatial dimension are extracted in the same way as those in the channel compression convolution layer for frame features, but the weight parameters are initialized independently. Through convolution, a spatial feature map of the frame features, i.e., the first spatial feature map, is output. The shape is [N, 32, 32, 1], representing the static anomaly importance of each pixel, where N is the batch size, 32×32 is the spatial dimension, and 1 is the number of channels. Simultaneously, a spatial feature map of the frame difference features, i.e., the second spatial feature map, is output. The shape is [N, 32, 32, 1], which represents the dynamic abnormal importance of each pixel.

[0049] Step A22: Input the first spatial feature map and the second spatial feature map into the activation layer for activation, respectively, to obtain the first activation feature map and the second activation feature map; In practice, the activation layer is a Sigmoid activation layer, which can be used to first activate the Sigmoid activation layer. Apply sigmoid activation to generate the first activation feature map. The shape is [N, 32, 32, 1]. The higher the value, the more likely the corresponding pixel is to be a static abnormal region.

[0050] Then, the Sigmoid activation layer is used to... Apply sigmoid activation to generate a second activation feature map. The shape is [N, 32, 32, 1], which represents the dynamic abnormal importance of each pixel.

[0051] Step A23: Perform abnormal region feature enhancement on the first activation feature map and the second activation feature map respectively to obtain the first attention feature map and the second attention feature map.

[0052] In practical implementation, the first activation feature map and frame features can be combined. Element-wise multiplication yields the enhanced frame features, i.e., the first attention feature map, calculated as follows: In the above formula, This is the first attention feature map.

[0053] In practical implementation, the second activation feature map and frame difference feature can be combined. Element-wise multiplication yields the enhanced frame difference features, i.e., the second attention feature map, calculated as follows: In the above formula, This is the second attention feature map.

[0054] Step S50: Perform feature fusion on the first attention feature map and the second attention feature map to generate fused features.

[0055] In practical implementation, a splicing and fusion method can be used to splice the first attention feature map and the second attention feature map along the channel dimension. Assuming the shape of the first attention feature map is [N, 32, 32, C1] and the shape of the second attention feature map is [N, 32, 32, C2], the fused feature shape obtained after splicing is [N, 32, 32, C1+C2]. This splicing method can retain the information of each of the two attention feature maps, so that the fused feature simultaneously contains feature information of both static and dynamic abnormal regions in the video frame.

[0056] Besides concatenation fusion, weighted fusion can also be used. First, weight coefficients are assigned to the first and second attention feature maps, which can be allocated based on the importance of static and dynamic anomalies in the actual application scenario. For example, in some scenarios, dynamic anomalies may be more critical, so a higher weight coefficient can be assigned to the second attention feature map. Weighted fusion allows for flexible adjustment of the contribution levels of static and dynamic anomaly features according to actual needs, further improving the accuracy and adaptability of anomaly behavior recognition. Whether concatenation fusion or weighted fusion, the final fused features provide richer and more comprehensive feature information for subsequent anomaly behavior classification and recognition, helping to improve the performance of the entire anomaly behavior recognition system.

[0057] Step S60: Perform abnormal behavior discrimination on the fusion feature to obtain an abnormal behavior recognition result.

[0058] It can be understood that the discrimination module can perform abnormal behavior discrimination on the fusion feature to obtain the abnormal behavior probability, and determine the final abnormal behavior recognition result through the abnormal behavior probability. The abnormal behavior recognition result may include the existence or non - existence of abnormal behavior. After obtaining the abnormal behavior recognition result, the abnormal behavior recognition result can be output through the abnormal result output module.

[0059] In a feasible implementation manner, step S60 may include steps A31 - A34: Step A31: Perform dimensionality compression on the fusion feature to obtain a global feature; It should be noted that the discrimination process of the fusion feature is four steps: feature compression - fully connected mapping - probability output - threshold judgment. The core is a lightweight fully connected network, that is, MLP and threshold comparison logic. Therefore, the fusion feature can be first dimensionally compressed to eliminate the spatial dimension, extract channel - level statistical features, and obtain the global feature. The calculation is as follows: In the above formula, is the global feature, retaining the channel - level abnormal information of the fusion feature. GAP is the global average pooling operation, H×W is the spatial size of the fusion feature, used to compress the spatial dimension H×W to 1×1, and the output shape is [N, C1 + C2], where N is the batch size and C1 + C2 is the number of channels. Through dimensionality compression, the computational amount can be reduced and more representative global features can be extracted, providing an efficient input for subsequent classification.

[0060] Step A32: Perform dimensionality reduction processing and non - linear transformation processing on the global feature to obtain a hidden feature; In specific implementation, the dimensionality reduction processing and non - linear transformation processing of the global feature can be performed through a fully connected layer and a LeakyReLU activation layer. The fully connected layer maps the global feature to a low - dimensional space through a weight matrix and outputs the abnormal behavior probability. Assuming the dimension of the global feature is C1 + C2, the output dimension of the fully connected layer is set to D, D < C1 + C2, and the shape of the weight matrix is [C1 + C2, D]. Dimensionality compression is achieved through matrix multiplication. Subsequently, a LeakyReLU activation layer is connected, and its negative half - axis slope is set to 0.01 to enhance the non - linear expression ability while maintaining the gradient flow. The processed hidden feature has a shape of [N, D], where N is the batch size and D is the hidden layer dimension. This step reduces the number of parameters through dimensionality reduction and prevents over - fitting. The non - linear transformation enables the model to learn more complex feature interaction patterns and provides a more discriminative feature representation for subsequent classification.

[0061] Step A33: Map the hidden features to abnormal behavior to obtain the abnormal probability of the abnormal behavior; In practical implementation, a fully connected mapping layer can be used to map hidden features to abnormal behavior. This layer maps the hidden features from a low-dimensional space to the target class space, outputting the raw score for each class. Assuming the number of abnormal behavior classes is K, and the weight matrix of the fully connected layer has a shape of [D, K], matrix multiplication transforms the hidden features into a raw score matrix of shape [N, K]. A Softmax activation layer is then applied to normalize the raw scores, obtaining the probability value corresponding to each class, with the sum of all probabilities being 1. The final output abnormal probability vector has a shape of [N, K], where the k-th element represents the probability that the sample belongs to the k-th class of abnormal behavior. This step achieves the transformation from the feature space to the class space through the fully connected layer, while the Softmax activation layer converts the output into a probability distribution, providing a basis for subsequent threshold judgment.

[0062] Step A34: Obtain the abnormal behavior identification result based on the abnormal probability.

[0063] It is understandable that a threshold can be set in advance, so that the abnormal probability is compared with the threshold to obtain the abnormal behavior identification result. Therefore, step A34 may include: obtaining a preset probability threshold; comparing the abnormal probability with the preset probability threshold; and determining that the abnormal behavior result is that there is abnormal behavior when the abnormal probability is greater than or equal to the preset probability threshold.

[0064] In practical implementation, the preset probability threshold should be set according to the actual application scenario and model performance requirements. For example, in security monitoring scenarios, it can be set to 0.9 to reduce the false alarm rate, while in industrial inspection scenarios, it can be appropriately reduced to 0.7 to improve sensitivity. When the maximum value in the anomaly probability vector is greater than or equal to the threshold, the system determines that the current video segment has abnormal behavior and records the corresponding timestamp, anomaly type, and confidence score; if all probability values ​​are lower than the threshold, it is determined to be normal behavior. This discrimination process is implemented through a threshold comparison layer, whose input is the probability vector output by Softmax, and the output is a binary discrimination result. At the same time, it can generate a structured log containing anomaly location, category, and probability, providing data support for subsequent manual review.

[0065] like Figure 3 As shown, Figure 3The diagram below illustrates the system architecture of this application, including an input preprocessing module, a bimodal feature extraction module, an anomaly attention module, an adaptive dynamic weighted fusion module, and a discriminant output module. The input preprocessing module extracts, normalizes, and resizes the original video sequence; it also calculates the inter-frame difference image between adjacent frames. The bimodal feature extraction module extracts features. The anomaly attention module is placed after the output of the convolutional temporal features of the bi-branch system to generate anomaly region attention maps. The adaptive dynamic weighted fusion module adaptively adjusts the weight ratios according to the features to perform feature fusion. The discriminant output module performs fully connected mapping and activation on the fused features and outputs the probability of abnormal behavior.

[0066] This embodiment provides an abnormal behavior recognition method that effectively improves the accuracy and robustness of abnormal behavior detection by integrating dual information from video frame sequences and inter-frame difference images. This method not only captures static features in the video but also captures dynamic change information through inter-frame difference images; the combination of these two aspects makes the recognition of abnormal behavior more comprehensive. Furthermore, by introducing an attention mechanism to enhance features in key regions, the model's sensitivity to abnormal behavior is further improved, reducing false positives and false negatives. Finally, through feature fusion and abnormal behavior discrimination, the method can quickly and accurately provide abnormal behavior recognition results, offering strong technical support for video surveillance, security protection, and other fields.

[0067] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30 includes steps S301 to S303: Step S301: Input the video frame sequence and the video frame difference sequence into a distributed convolutional network to extract spatial features, thereby obtaining video frame spatial features and video frame difference spatial features.

[0068] It should be noted that the bimodal feature extraction module includes two independent spatiotemporal feature branches: a frame branch and a frame difference branch. The frame branch employs a spatiotemporally distributed convolutional network and a separable convolutional long short-term memory network, namely TimeDistributed CNN and SepConvLSTM. The structure of the frame difference branch is similar to that of the frame branch. Therefore, the video frame sequence can be input into the frame branch, and the video frame difference sequence can be input into the frame difference branch. Spatial features are extracted first by inputting the video frame sequence into the spatiotemporally distributed convolutional network in the frame branch to extract spatial features, thus obtaining static spatial features, i.e., video frame spatial features. By inputting the video frame difference sequence into the spatiotemporally distributed convolutional network in the frame difference branch to extract spatial features, the dynamic features of inter-frame changes are extracted, thus obtaining the video frame difference spatial features.

[0069] By applying CNN layers to each frame in a video frame sequence, the temporal dimension is preserved. Specifically, the CNN structure contains three convolutional blocks. By convolving each frame in the video frame sequence, the spatial feature shape is output, thus obtaining the video frame spatial features. Similarly, the video frame difference spatial features can be obtained.

[0070] Step S302: Input the video frame spatial features and the video frame difference spatial features into a separate convolutional long short-term memory network for temporal feature extraction to obtain video frame temporal features and video frame difference temporal features.

[0071] It is understandable that after extracting the spatial features, the video frame spatial features can be input into the split convolutional long short-term memory network in the frame branch for temporal feature extraction, and the video frame difference spatial features can be input into the split convolutional long short-term memory network in the frame difference branch for temporal feature extraction, thereby obtaining the video frame temporal features and the video frame difference temporal features respectively.

[0072] Step S303: Obtain the frame features of the video frame sequence based on the video frame temporal features, and obtain the frame difference features of the video frame difference sequence based on the video frame difference temporal features.

[0073] In practice, the output of the last time step of the decoupled convolutional long short-term memory network can be taken as the final feature of the frame branch and the frame difference branch, thereby obtaining the frame features of the video frame sequence and the frame difference features of the video frame difference sequence.

[0074] This embodiment extracts spatial features using a temporally distributed convolutional network, effectively extracting detailed information and important features from video frames, thereby improving the model's understanding of video content. Spatial features can capture visual information about static objects, backgrounds, and important dynamic areas. Using a separable convolutional long short-term memory network for temporal feature extraction can effectively identify patterns of dynamic changes in the video. Long short-term memory networks excel at processing time-series data, capturing temporal dependencies between video frames, and identifying temporal behaviors such as motion trajectories and event sequences, which is particularly important for the analysis of moving objects. Feature extraction using separable convolution reduces model complexity, making it easier to generalize and less prone to overfitting. This is especially important when processing video data, as video data often contains a large amount of redundant information. The combination of temporally distributed convolutional networks and separable convolutional long short-term memory networks optimizes the utilization of computational resources, improves frame processing speed, and enables the system to perform real-time analysis and response more efficiently. By extracting features from video frames and frame differences separately, the events and changes occurring in the video can be understood more clearly, thereby improving the model's interpretability. Frame difference features can highlight motion changes, thereby helping to identify action or state changes.

[0075] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Step S50 includes steps S501 to S503: Step S501: Perform spatial feature compression on the first attention feature map and the second attention feature map respectively to obtain the first channel-level statistical features and the second channel-level statistical features.

[0076] It should be noted that feature fusion can be achieved through an adaptive dynamic weighted fusion module. This module achieves feature fusion through four steps: global feature statistics, vector concatenation, adaptive weight generation, and dynamic weighting. The core of this module is a lightweight fully connected network that generates dynamic weights. Therefore, spatial feature compression (i.e., global average pooling) can be performed on the first and second attention feature maps respectively to eliminate spatial dimensions and extract channel-level statistical features, resulting in the first and second channel-level statistical features.

[0077] The first channel-level statistical characteristic is GAP ( The second channel-level statistical characteristic is GAP ( ).

[0078] Step S502: Concatenate the feature vectors of the first channel-level statistical features and the second channel-level statistical features to obtain concatenated statistical features.

[0079] In practice, two channel-level statistical features can be concatenated along the channel dimension to form a concatenated statistical feature, calculated as follows: In the above formula, To construct statistical features, the statistical features are combined with the statistical features of static and dynamic anomalies.

[0080] Step S503: Perform feature fusion based on the splicing statistical features to generate fused features.

[0081] In practical implementation, feature fusion can be performed on the spliced ​​statistical features through a multilayer perceptron (MLP) to generate fused features. The MLP consists of two layers of perceptrons: the first fully connected layer has an input dimension of 512, an output dimension of 128, an activation function of ReLU, weight initialization of He normal, and a bias of 0; the second fully connected layer has an input dimension of 128, an output dimension of 2, an activation function of Softmax, weight initialization of Xavier normal, and a bias of 0.

[0082] In one feasible implementation, step S503 may include steps B11-B12: Step B11: Input the spliced ​​statistical features into the multilayer perceptron to generate normalized weights, the normalized weights including a first weight and a second weight. It should be noted that the spliced ​​statistical features can be input into the MLP structure, and the output is two normalized weight values. Specifically, the weighted fusion ratio of the two branches can be adaptively adjusted according to the features of the video segment.

[0083] The normalized weights are generated as follows: In the above formula, To normalize the weights, =[ 1, 2], satisfying the first weight 1+Second Weight 2=1, To splice statistical features.

[0084] Step B12: Perform feature fusion based on the first weight, the second weight, the first attention feature map, and the second attention feature map to generate fused features.

[0085] Understandably, after determining the weights, they can be expanded to a spatial feature-compatible dimension, and the enhanced features can be weighted and summed. Therefore, the first attention feature map and the second attention feature map can be fused using the weights to generate fused features, as shown in the following formula: Feature fusion is performed using the normalized weights in the above formula to obtain the fused features. It can achieve a dynamic balance between frame features and differential features in different scenarios.

[0086] like Figure 6 As shown, Figure 6 This is a schematic diagram of the adaptive weighted fusion module. The module achieves adaptive fusion of dual-branch features through weight generation and dynamic weighting, solving the scene adaptation problem between static and dynamic anomalies. The input is the frame branch attention feature map. +Frame Difference Branch Attention Feature Map Consistent dimensions ensure matching weights. Weight generation network: Global average pooling (GAP) is performed on the dual-branch attention feature maps to extract channel-level statistical vectors GAP(…). ) and GAP The concatenated vectors are input into a two-layer MLP, i.e., input 512 → hidden 128 → output 2, and then normalized weights are generated by Softmax activation. and , 1 represents the frame branch weight. The weights are the frame difference branch weights. Weighted fusion: The weights are expanded to a spatially compatible [N,1,1,1], element-wise multiplied with the dual-branch features, and then summed to output the fused features. .

[0087] This embodiment effectively extracts the most relevant statistical features for the task by spatially compressing the attention feature map. This selective aggregation of information enhances the model's focus on key features, thereby strengthening its performance on specific tasks. By concatenating feature vectors, not only is important information retained from the first and second attention feature maps, but the feature dimensionality is also increased. This concatenation operation allows the model to learn diverse information from different feature maps, contributing to a more comprehensive feature representation. By fusing the statistical information of different attention features, the model can better capture complex patterns and relationships, especially when dealing with tasks with multiple important information sources. This capability makes the model more flexible in recognizing and parsing multiple features.

[0088] For example, to help understand the implementation flow of the abnormal behavior recognition method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 7 , Figure 7 A simplified flowchart of an abnormal behavior recognition method is provided, specifically including: Video sequence input: receiving the original video stream or file as the starting point for system processing. Frame difference calculation: calculating the absolute difference between consecutive adjacent frames to generate a frame difference sequence. Feature extraction: dual-modal parallel processing, frame branch: extracting static spatial features from the frame sequence using a CNN, then modeling temporal changes using a SepConvLSTM to output frame branch features; frame difference branch: extracting dynamic spatiotemporal features from the frame difference sequence using the same CNN+SepConvLSTM, outputting frame difference branch features. Anomaly attention enhancement: generating an attention map through an anomaly attention module, weighting the dual-branch features element-wise to enhance the response to abnormal regions, such as stationary abnormal objects and rapidly moving areas. Adaptive weighted fusion: based on the global statistical information of the dual-branch enhanced features, generating dynamic weights through a lightweight MLP, and performing weighted summation of the features. Anomaly detection and judgment: mapping the fused features to anomaly probabilities through a fully connected network, activating Sigmoid, and combining with a threshold T to determine the result; a probability ≥ T indicates an anomaly.

[0089] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the abnormal behavior identification method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0090] This application also provides an abnormal behavior recognition device, please refer to... Figure 8 The abnormal behavior identification device includes: The acquisition module 10 is used to acquire the original video sequence and perform frame extraction, size adjustment and normalization on the original video sequence to obtain a video frame sequence. Calculation module 20 is used to calculate the inter-frame difference image between adjacent video frames in the video frame sequence to obtain a video frame difference sequence; The feature extraction module 30 is used to extract features from the video frame sequence and the video frame difference sequence respectively, to obtain the frame features of the video frame sequence and the frame difference features of the video frame difference sequence. Feature enhancement module 40 is used to enhance the abnormal region features of the frame features and the frame difference features respectively, and generate a first attention feature map and a second attention feature map; Feature fusion module 50 is used to fuse the first attention feature map and the second attention feature map to generate fused features; The discrimination module 60 is used to discriminate abnormal behavior based on the fused features and obtain abnormal behavior identification results.

[0091] The abnormal behavior recognition device provided in this application, employing the abnormal behavior recognition method in the above embodiments, can solve the technical problems of poor adaptability to complex environments and lack of time dynamic change perception capability in existing abnormal behavior recognition methods. Compared with the prior art, the beneficial effects of the abnormal behavior recognition device provided in this application are the same as those of the abnormal behavior recognition method provided in the above embodiments, and other technical features in the abnormal behavior recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0092] This application provides an abnormal behavior recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the abnormal behavior recognition method in the above embodiment 1.

[0093] The following is for reference. Figure 9 The diagram illustrates a structural schematic suitable for implementing an abnormal behavior recognition device according to embodiments of this application. The abnormal behavior recognition device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.Figure 9 The abnormal behavior recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0094] like Figure 9 As shown, the abnormal behavior recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the abnormal behavior recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the abnormal behavior identification device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show abnormal behavior identification devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0095] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0096] The abnormal behavior recognition device provided in this application, employing the abnormal behavior recognition method in the above embodiments, can solve the technical problems of poor adaptability to complex environments and lack of time dynamic change perception capability in existing abnormal behavior recognition methods. Compared with the prior art, the beneficial effects of the abnormal behavior recognition device provided in this application are the same as those of the abnormal behavior recognition method provided in the above embodiments, and other technical features in this abnormal behavior recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0097] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0099] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the abnormal behavior identification method in the above embodiments.

[0100] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0101] The aforementioned computer-readable storage medium may be included in the abnormal behavior recognition device; or it may exist independently and not assembled into the abnormal behavior recognition device.

[0102] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the abnormal behavior recognition device, the abnormal behavior recognition device causes the following: it acquires an original video sequence and performs frame extraction, resizing, and normalization on the original video sequence to obtain a video frame sequence; it calculates the inter-frame difference image between adjacent video frames in the video frame sequence to obtain a video frame difference sequence; it extracts features from the video frame sequence and the video frame difference sequence respectively to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence; it enhances the frame features and the frame difference features for abnormal regions respectively to generate a first attention feature map and a second attention feature map; it fuses the first attention feature map and the second attention feature map to generate a fused feature; and it performs abnormal behavior discrimination on the fused feature to obtain an abnormal behavior recognition result.

[0103] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0106] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described abnormal behavior recognition method. This solves the technical problems of poor adaptability to complex environments and lack of time-based dynamic change perception capabilities in existing abnormal behavior recognition methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the abnormal behavior recognition method provided in the above embodiments, and will not be repeated here.

[0107] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the abnormal behavior recognition method described above.

[0108] The computer program product provided in this application can solve the technical problems of poor adaptability to complex environments and lack of ability to perceive dynamic changes over time in existing abnormal behavior recognition methods. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the abnormal behavior recognition method provided in the above embodiments, and will not be repeated here.

[0109] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An abnormal behavior recognition method characterized by comprising: The abnormal behavior recognition method comprises: An original video sequence is acquired, and frame extraction, size adjustment and normalization are performed on the original video sequence to obtain a video frame sequence; An inter-frame difference image between adjacent video frames in the video frame sequence is calculated to obtain a video frame difference sequence; Feature extraction is respectively performed on the video frame sequence and the video frame difference sequence to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence; Abnormal region feature enhancement is respectively performed on the frame features and the frame difference features to generate a first attention feature map and a second attention feature map; Feature fusion is performed on the first attention feature map and the second attention feature map to generate a fusion feature; Abnormal behavior discrimination is performed on the fusion feature to obtain an abnormal behavior recognition result.

2. The method of claim 1, wherein, The step of performing feature extraction on the video frame sequence and the video frame difference sequence respectively to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence comprises: The video frame sequence and the video frame difference sequence are respectively input into a time distribution convolution network to extract spatial features to obtain video frame spatial features and video frame difference spatial features; The video frame spatial features and the video frame difference spatial features are respectively input into a separation convolution long short-term memory network for time sequence feature extraction to obtain video frame time sequence features and video frame difference time sequence features; The frame features of the video frame sequence are obtained according to the video frame time sequence features, and the frame difference features of the video frame difference sequence are obtained according to the video frame difference time sequence features.

3. The method of claim 1, wherein, The step of performing feature fusion on the first attention feature map and the second attention feature map to generate a fusion feature comprises: Spatial feature compression is respectively performed on the first attention feature map and the second attention feature map to obtain first channel level statistical features and second channel level statistical features; Feature vector splicing is performed on the first channel level statistical features and the second channel level statistical features to obtain spliced statistical features; Feature fusion is performed according to the spliced statistical features to generate a fusion feature.

4. The method of claim 3, wherein, The step of performing feature fusion according to the spliced statistical features to generate a fusion feature comprises: The spliced statistical features are input into a multi-layer perceptron to generate normalized weights, wherein the normalized weights comprise first weights and second weights; Feature fusion is performed according to the first weights, the second weights, the first attention feature map and the second attention feature map to generate a fusion feature. The step of performing abnormal region feature enhancement on the frame features and the frame difference features respectively to generate a first attention feature map and a second attention feature map comprises:

5. The method of claim 1, wherein, The frame features and the frame difference features are respectively input into a channel compression convolution layer for convolution to obtain a first spatial feature map and a second spatial feature map; The first spatial feature map and the second spatial feature map are respectively input into an activation layer for activation to obtain a first activation feature map and a second activation feature map; The first activation feature map and the second activation feature map are respectively subjected to abnormal region feature enhancement to obtain a first attention feature map and a second attention feature map. ​ 6. The method of claim 1, wherein, The step of performing abnormal behavior discrimination on the fusion feature to obtain an abnormal behavior recognition result comprises: dimensional compression is performed on the fusion feature to obtain a global feature; dimensional reduction processing and nonlinear transformation processing are performed on the global feature to obtain a hidden feature; abnormal behavior mapping is performed on the hidden feature to obtain an abnormal probability of abnormal behavior; an abnormal behavior recognition result is obtained according to the abnormal probability.

7. The method of claim 6, wherein, The step of obtaining an abnormal behavior recognition result according to the abnormal probability comprises: a preset probability threshold is obtained; the abnormal probability is compared with the preset probability threshold; when the abnormal probability is greater than or equal to the preset probability threshold, it is determined that the abnormal behavior result is that there is abnormal behavior.

8. The method of any one of claims 1 to 7, wherein, The step of obtaining an original video sequence, performing frame extraction, size adjustment, and normalization on the original video sequence to obtain a video frame sequence comprises: an original video sequence, a preset sampling strategy, and a target size are obtained; frame extraction is performed on the original video sequence through the preset sampling strategy to obtain extracted video frames; channel conversion is performed on the extracted video frames to obtain converted video frames; the converted video frames are adjusted to the target size to obtain size-adjusted video frames; the size-adjusted video frames are normalized to obtain a video frame sequence.

9. An abnormal behavior recognition apparatus characterized by comprising: The device comprises: an acquisition module configured to obtain an original video sequence, perform frame extraction, size adjustment, and normalization on the original video sequence, and obtain a video frame sequence; a calculation module configured to calculate interframe difference images between adjacent video frames in the video frame sequence to obtain a video frame difference sequence; a feature extraction module configured to perform feature extraction on the video frame sequence and the video frame difference sequence respectively to obtain frame features of the video frame sequence and frame difference features of the video frame difference sequence; a feature enhancement module configured to perform abnormal region feature enhancement on the frame features and the frame difference features respectively to generate a first attention feature map and a second attention feature map; a feature fusion module configured to perform feature fusion on the first attention feature map and the second attention feature map to generate a fusion feature; a discrimination module configured to perform abnormal behavior discrimination on the fusion feature to obtain an abnormal behavior recognition result.

10. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the steps of the abnormal behavior recognition method according to any one of claims 1 to 8 are implemented.

Images