A park abnormal behavior few-sample real-time monitoring method and system

Abstract

The application provides a kind of park abnormal behavior few sample real-time monitoring method and system, it is related to computer technology field, the method includes: obtaining park real-time monitoring video stream as query set, and from preset abnormal behavior database, call several known category video samples as support set;Through feature extraction network, video frame sequence in query set and support set are processed respectively, and spatial feature representing static details, timing feature representing global evolution and action feature representing interframe change intensity are extracted in parallel;Based on the motion intensity represented by action feature, the spatial feature is weighted and enhanced, and the enhanced spatial feature is fused with timing feature to generate comprehensive feature;The comprehensive feature of query set is input into matching model as query item, the comprehensive feature of support set is adaptively reweighted, and based on the query item and the reweighted support set comprehensive feature, it is judged whether there is abnormal behavior in park real-time monitoring video stream.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for real-time monitoring of abnormal behavior in a park with a small sample size. Background Technology

[0002] Computer vision-based intelligent monitoring systems are playing an increasingly important role in public safety. Action recognition, as a core technology of intelligent monitoring, aims to automatically analyze and identify abnormal behavior from video streams. Traditional action recognition methods are mostly based on deep learning (such as C3D, Two-Stream, and I3D network architectures). While these models perform well on specific datasets, they typically require massive amounts of labeled data for training. However, in real-world park monitoring scenarios, abnormal behaviors (such as sudden events) often occur infrequently, and obtaining samples is difficult, making it challenging to collect sufficient training data. Therefore, how to achieve high-precision action recognition with only a small number of samples (i.e., few-sample action recognition) has become a critical problem urgently needing to be solved by industry and academia.

[0003] Currently, research on few-shot action recognition mainly focuses on metric learning-based methods. These methods typically map video samples to a feature space and classify them by calculating the distance between the query set and the support set. Early approaches, such as Prototypical Networks, used the average features of each class as prototypes for matching; more advanced approaches, such as TRX (Temporal Relational CrossTransformer) and ACTION-Net, began to explore using temporal alignment algorithms or constructing simple spatiotemporal features to capture dynamic information in videos. Specifically, existing techniques typically employ a unified backbone network to extract video frame features, and then use a pre-defined distance function (such as Euclidean distance or cosine similarity) to measure the similarity between samples.

[0004] However, the aforementioned existing technologies still have limitations when dealing with complex abnormal behavior monitoring in industrial parks: existing metric learning methods often adopt a static metric approach in the feature matching stage, meaning that the feature representation of the support set is fixed and cannot be adaptively adjusted according to the characteristics of the sample to be identified (query set). This static matching mechanism is difficult to capture the complex spatiotemporal evolution relationships in video actions, especially when facing unseen abnormal behaviors with high intra-class variability. The model cannot effectively establish deep semantic relationships between query samples and support samples, resulting in insufficient accuracy in action recognition and failing to meet the needs of real-time and accurate monitoring. Summary of the Invention

[0005] This invention provides a method and system for real-time monitoring of abnormal behavior in a park with few samples, which solves the defects of rigid matching mechanism and insufficient feature expression in the existing technology in the recognition of action with few samples.

[0006] This invention provides a method for real-time monitoring of anomalous behavior in a park with a small sample size, comprising:

[0007] The system obtains real-time monitoring video streams from the park as a query set and retrieves video samples of several known categories from a pre-set abnormal behavior database as a support set.

[0008] The video frame sequences in the query set and the support set are processed by a pre-trained feature extraction network to extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0009] Based on the motion intensity represented by the action features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the temporal features to generate comprehensive features;

[0010] The comprehensive features of the query set are input into the pre-trained matching model as the query term. The comprehensive features of the support set are adaptively reweighted. Based on the query term and the reweighted comprehensive features of the support set, it is determined whether there is abnormal behavior in the real-time monitoring video stream of the park.

[0011] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0012] According to the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention, the feature extraction network includes a channel attention unit;

[0013] The extraction of motion features representing the intensity of inter-frame changes specifically includes: obtaining the frame-level feature vector output by the feature extraction network for each frame in the video frame sequence; performing differential calculation on the frame-level feature vectors of adjacent frames to obtain the original motion features; calculating the importance weight of each feature channel in the original motion features through the channel attention unit; mapping the importance weight to a preset value range through an activation function, and using the mapped weights to filter the original motion features to suppress background noise interference and retain key motion information, and using the filtered features as the motion features.

[0014] According to the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention, the feature extraction network includes a spatial attention unit and a channel attention unit;

[0015] The extraction of spatial features representing static details specifically includes: obtaining the frame-level feature vector output by the feature extraction network for each frame in the video frame sequence; extracting salient target region information from the frame-level feature vector through the spatial attention unit; recalibrating the feature channels of the feature vector after extracting salient target region information through the channel attention unit, and outputting the spatial features.

[0016] According to the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention, the feature extraction network includes multiple stacked self-attention layers;

[0017] The extraction of temporal features representing global evolution specifically includes: obtaining the frame-level feature vectors output by the feature extraction network for each frame in the video frame sequence, and constructing a feature sequence set by arranging the frame-level feature vectors output by each frame in the video frame sequence in chronological order; performing global context modeling on the feature sequence set through the self-attention layer to capture long-distance dependencies between frames, and outputting the temporal features.

[0018] According to the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention, the spatial features are weighted and enhanced based on the motion intensity represented by the action features, and the enhanced spatial features are fused with the temporal features to generate comprehensive features, specifically including:

[0019] Using the action features as weight vectors, element-wise multiplication is performed with the spatial features to obtain the action-enhanced spatial features; the action-enhanced spatial features are then residually concatenated with the unprocessed original spatial features; the feature vector resulting from the residual concatenation is then concatenated or added with the temporal features to obtain the comprehensive features.

[0020] According to the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention, the comprehensive features of the query set are input into a pre-trained matching model as query terms, and the comprehensive features of the support set are adaptively reweighted, specifically including:

[0021] Using the pre-trained matching model, the generated comprehensive features corresponding to the support set are linearly mapped to key vectors and value vectors, and the generated comprehensive features corresponding to the query set are linearly mapped to query vectors. The dot product similarity between the query vector and the key vector is calculated and normalized to generate an attention weight map. The value vector is weighted and aggregated using the attention weight map to obtain the reweighted comprehensive features of the support set.

[0022] According to the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention, the step of determining whether there is abnormal behavior in the real-time monitoring video stream of the park based on the comprehensive features of the query item and the reweighted support set specifically includes:

[0023] For each action category in the support set, the action prototype of that category is determined based on the comprehensive features of the support set after reweighting of all video samples in that category; the distance between the query item and the action prototype of each action category is calculated to obtain the class distance of the query item for each action category; the action category corresponding to the smallest class distance is selected as the abnormal behavior category of the real-time monitoring video stream of the park.

[0024] This invention also provides a real-time monitoring system for anomalous behavior in a park with a small sample size, comprising:

[0025] The data acquisition module is used to acquire real-time monitoring video streams of the park as a query set, and retrieve video samples of several known categories from a pre-set abnormal behavior database as a support set.

[0026] The feature extraction module is used to process the video frame sequences in the query set and the support set respectively through a pre-trained feature extraction network, and extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0027] The feature fusion module is used to weight and enhance the spatial features based on the motion intensity represented by the action features, and then fuse the enhanced spatial features with the temporal features to generate a comprehensive feature;

[0028] The anomaly detection module is used to input the comprehensive features of the query set into a pre-trained matching model as a query term, adaptively reweight the comprehensive features of the support set, and determine whether there is abnormal behavior in the real-time monitoring video stream of the park based on the query term and the reweighted comprehensive features of the support set.

[0029] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the small sample real-time monitoring method for abnormal behavior in a park as described above.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the small sample real-time monitoring method for abnormal behavior in a park as described above.

[0032] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for real-time monitoring of abnormal behavior in a park with few samples as described above.

[0033] The present invention provides a method and system for real-time monitoring of abnormal behavior in a park with few samples. It utilizes action features representing the intensity of inter-frame changes to weight and enhance spatial features representing static details, and deeply integrates these features with temporal features. This effectively suppresses environmental noise interference in complex park environments and highlights the semantic expression of key action regions. Furthermore, a pre-trained matching model uses the comprehensive features of the query set as query terms to adaptively reweight the comprehensive features of the support set. This changes the static measurement method in traditional prototype networks where support set features are fixed once extracted. Instead, it enables dynamic adjustment of the support set prototype features based on the characteristics of the current sample to be identified, thus establishing a deeper semantic association between the query set and the support set. Ultimately, with only a few reference samples, it significantly improves the system's accuracy in identifying complex abnormal behavior within the park and its robustness in real-time monitoring. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is one of the flowcharts of the method for real-time monitoring of abnormal behavior in a park with few samples provided by the present invention.

[0036] Figure 2 This is a schematic diagram of the process for extracting spatial features that characterize static details, provided by the present invention.

[0037] Figure 3 This is a schematic diagram of the process for extracting motion features that characterize the intensity of inter-frame changes, provided by the present invention.

[0038] Figure 4 This is a flowchart illustrating the process of extracting temporal features representing global evolution provided by the present invention.

[0039] Figure 5 This is a schematic diagram of the process for generating comprehensive features provided by the present invention.

[0040] Figure 6 This is a flowchart illustrating the process of identifying abnormal behavior in videos provided by the present invention.

[0041] Figure 7This is a flowchart illustrating the process of determining the category of abnormal behavior in a real-time monitoring video stream of a park, provided by the present invention.

[0042] Figure 8 This is the second flowchart illustrating the process of determining the category of abnormal behavior in a real-time monitoring video stream of a park, provided by the present invention.

[0043] Figure 9 This is a schematic diagram of the structure of the real-time monitoring system for abnormal behavior in a park with few samples provided by the present invention.

[0044] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] The background section points out that existing few-sample action recognition methods are mainly limited by static metric matching mechanisms, making it impossible to adaptively adjust support set features based on query samples. However, in actual park abnormal behavior monitoring scenarios, in addition to the limitations of the matching mechanism, the purity of feature extraction and the interaction mode of multi-dimensional features are also key bottlenecks restricting recognition accuracy, specifically in the following two aspects:

[0047] First, regarding motion information extraction and denoising. To meet the real-time requirements of park monitoring, existing technologies (such as TSN) often abandon computationally intensive optical flow methods and instead use the computationally simpler RGB frame difference method to capture motion information. However, the simple frame difference method is extremely sensitive to the environment; common background noise in parks, such as swaying leaves, changes in light and shadow, and the movement of unrelated people in the distance, can produce significant pixel differences. Existing technologies lack effective channel filtering mechanisms, often mistaking this background noise for core motion features, leading to a large number of false alarms when the model judges subtle movements.

[0048] Secondly, regarding the depth of spatiotemporal feature fusion, traditional two-stream networks or 3D convolutional networks often employ simple concatenation or addition operations when fusing spatial and temporal features. This approach ignores the inherent logical connection between action intensity and image detail—in reality, the intensity of the action often determines which spatial regions in the image (such as a waving arm or a falling torso) are worth paying attention to. The lack of this interactive fusion mechanism for action-guided space makes it difficult for the model to accurately locate the subject of the action in complex backgrounds.

[0049] In view of the technical defects of the prior art, such as severe background noise interference, fragmented spatiotemporal feature interaction, and rigid matching mechanism, this application provides a method for real-time monitoring of abnormal behavior in a park based on few-sample learning.

[0050] The core concept of this invention lies in constructing a multi-dimensional feature extraction network that refines and optimizes features, coupled with a dynamically adaptable feature matching model. Specifically, this embodiment first uses a channel attention mechanism to adaptively filter frame difference features, suppressing environmental noise at its source; secondly, it innovatively uses action intensity features as weights to enhance spatial features, achieving deep fusion that "contrasts movement with stillness"; finally, it uses the Transformer architecture to establish a dynamic association between the query set and the support set, achieving accurate recognition under conditions of few samples.

[0051] Before describing the technical solutions of the embodiments of the present invention, the terminology used in the embodiments of the present invention will be explained illustratively.

[0052] Transformer: A model that uses attention mechanisms to solve natural language translation problems.

[0053] Deep learning: A method in machine learning based on learning representations of data.

[0054] Convolutional Neural Networks: Convolutional Neural Networks are a type of feedforward neural network that includes convolutional computation and has a deep structure. They are one of the representative algorithms of deep learning.

[0055] Few-shot learning: A learning method that learns from a small number of labeled samples, thereby enabling the model to recognize new categories.

[0056] Few-shot action recognition: A method based on few-shot learning to identify and understand different action categories from a limited number of action data samples.

[0057] ResNet50: A neural network architecture often used for image feature generation.

[0058] Channel attention mechanism: a technique used to enhance the performance of neural networks. Its core idea is to enable the network to automatically learn and emphasize the feature channels that are more important to the task, thereby improving the expressive power of the model.

[0059] Sigmoid layer: A type of activation function layer in a neural network, whose main function is to map the input value to an output value between 0 and 1.

[0060] Support set: A set of samples used to train the model to learn features.

[0061] Query set: The set of samples used for model prediction.

[0062] Action Prototype: In prototype learning, a prototype is typically a set of representative samples selected from the training data, with each prototype representing a category or cluster. A prototype can be the center of a data point (such as the mean vector) or a representative sample point.

[0063] L2 distance, also known as Euclidean distance, is a commonly used distance metric used to measure the distance between two vectors in a vector space. Given feature vectors x and y of length d, the L2 distance between x and y can be calculated using the following formula:

[0064] .

[0065] xi represents the component value of the eigenvector x in the i-th dimension, and yi represents the component value of the eigenvector y in the i-th dimension. The value of i ranges from 1 to d, where d is the dimension of the eigenvector.

[0066] Spatial features: Spatial features refer to the features extracted from video frames that represent static details. They are mainly used to describe static information such as the shape, texture, and color of objects in an image. In action recognition, spatial features can capture the static content of each frame in a video, providing basic information for subsequent action analysis. For example, in surveillance videos, spatial features can help identify the appearance features of objects (such as people and vehicles) in a scene.

[0067] Temporal features: Temporal features refer to features extracted from a sequence of video frames that characterize global evolution. They are mainly used to describe changes in objects or scenes in a video over time. Temporal features can capture the dynamic process of actions and are an indispensable part of action recognition. In surveillance videos, temporal features can help identify the trajectory of objects, the duration of actions, and so on.

[0068] Motion features: Motion features are features extracted from a video frame sequence that characterize the intensity of changes between frames. They are mainly used to describe the dynamic changes of objects or scenes over time. Motion features can highlight motion information in a video and are key to identifying abnormal behavior. In surveillance videos, motion features can help identify rapidly moving objects (such as fighting, falling, and other abnormal behaviors).

[0069] Raw motion features: Raw motion features are features obtained by differencing the feature vectors of adjacent frames, and are used to characterize the intensity of changes between frames. Raw motion features reflect the motion information of objects in a video, but may contain noise and background interference. In surveillance videos, raw motion features can help identify the direction and speed of an object's motion.

[0070] Salient Target Region Information: Salient target region information refers to the features extracted from video frames that represent salient regions of target objects. It is primarily used to highlight key target areas in the video. Salient target region information helps the model focus on important visual information while ignoring background noise.

[0071] Long-range dependencies: Long-range dependencies refer to the correlations between relatively distant frames in a video frame sequence. Capturing long-range dependencies is crucial for understanding dynamic processes in video, especially in action recognition.

[0072] Abnormal Behavior Categories: Abnormal behavior categories refer to the types of abnormal behavior that need to be identified in surveillance videos, such as fighting or falling. Identifying abnormal behavior categories is the core objective of the park's abnormal behavior monitoring system.

[0073] Feature extraction networks are specialized deep learning models designed to extract high-dimensional feature vectors from input data, such as images or video frames. These feature vectors characterize the semantic information of the input data, providing a foundation for subsequent tasks such as classification, detection, and recognition. In the real-time monitoring system for anomalous behavior in a park with few samples, the feature extraction network is responsible for extracting spatial, temporal, and action features from video frames.

[0074] The core function of feature extraction networks is to transform raw image or video frame data into semantically meaningful feature representations. These feature representations capture key information from the input data, such as the shape, texture, and motion of objects, thus supporting subsequent analysis and processing. In few-shot learning scenarios, the performance of feature extraction networks directly affects the model's generalization ability and recognition accuracy for new categories.

[0075] Feature extraction networks are typically based on convolutional neural network (CNN) architectures, with ResNet (Residual Network) being a common example. ResNet is a classic CNN architecture that addresses the vanishing gradient problem in deep network training by introducing residual connections. Different versions of ResNet (such as ResNet50 and ResNet101) perform well in image classification and feature extraction tasks. In this invention, ResNet50 is a commonly used feature extraction network with an output feature dimension of 2048×7×7.

[0076] Matching Model: A matching model is a deep learning model used to calculate the similarity between a query sample and a support set. Its core objective is to determine the category of the query sample or whether it matches a category in the support set through feature matching. In few-shot learning and meta-learning tasks, matching models are a key component for achieving fast classification and recognition.

[0077] In a real-time monitoring system for anomalous behavior in a park with few samples, the main role of the matching model is:

[0078] Feature matching: Calculate the similarity between the features of the query sample and the features of the supporting samples to determine the category to which the query sample belongs.

[0079] Dynamic adjustment: Based on the characteristics of the query sample, the features of the supporting samples are adaptively reweighted to enhance the model's ability to identify the query sample.

[0080] Classification decision: Based on the similarity calculation results, determine whether the query sample belongs to a certain known abnormal behavior category in the supporting samples, thereby realizing real-time monitoring of abnormal behavior.

[0081] Matching models are typically implemented based on the following core mechanisms:

[0082] Metric learning is the foundation of matching models. By learning a feature space, samples of the same class are closer together in that space, while samples of different classes are farther apart.

[0083] Self-Attention: By calculating the correlation weights between feature vectors, features are weighted and aggregated to capture long-distance dependencies between features.

[0084] Transformer architecture: The Transformer architecture is widely used in matching models, and its core is the multi-head self-attention mechanism. Transformers can effectively capture global dependencies between features and improve matching accuracy.

[0085] Prototypical Learning: Prototypical learning is a metric-based learning method that achieves classification by calculating the distance between the average features (prototype) of the supporting samples and the features of the query samples.

[0086] The real-time monitoring system for anomalous behavior in a park with a small sample size according to this invention can be deployed in various hardware environments. The specific configuration depends on the monitoring scale, real-time requirements, and data processing volume.

[0087] First, high-definition cameras or IP cameras are deployed in key locations within the campus to capture video streams in real time. These cameras need to have high resolution, low-light imaging capabilities, and a wide dynamic range to ensure high-quality video data. The captured video streams are transmitted to a data processing server via a local area network (LAN), wireless local area network (WLAN), or fiber optic network. 5G networks can be used for scenarios with high real-time requirements to reduce latency.

[0088] The data processing server is equipped with high-performance CPUs and GPUs to run feature extraction networks and matching models, rapidly processing video streams and performing anomaly behavior analysis. The server also features high-speed solid-state drives (SSDs) or distributed storage systems to ensure fast data read / write speeds and secure storage. In some scenarios, edge computing devices can be deployed near the cameras for preliminary data processing and feature extraction, reducing the amount of data transmitted to the central server and improving system real-time performance.

[0089] The monitoring workstation is equipped with a high-resolution display for real-time monitoring of video streams and displaying alarm information for abnormal behavior. It also features alarm devices such as sirens or SMS notification systems to promptly notify relevant personnel. In the system architecture, video streams captured by cameras are transmitted via network to a data processing server. The server-processed results are then transmitted back to the monitoring workstation for display and alarm activation. This hardware configuration offers high-performance processing capabilities, high real-time performance, scalability, and data security, enabling efficient real-time monitoring of a small sample size of abnormal behavior within the park and meeting the application needs of various scenarios.

[0090] The following is combined Figures 1-8 This invention describes a method for real-time monitoring of abnormal behavior in a park with a small sample size, as provided in an embodiment of the present invention.

[0091] Figure 1This is one of the flowcharts illustrating the method for real-time monitoring of abnormal behavior in a park with few samples provided in this embodiment of the invention, such as... Figure 1 As shown, the method includes the following:

[0092] Step 101: Obtain the real-time monitoring video stream of the park as the query set, and retrieve several video samples of known categories from the pre-set abnormal behavior database as the support set.

[0093] In a real-time monitoring system for anomalous behavior with a limited sample size in a park, data acquisition is fundamental to the entire process. First, the system collects real-time video streams from high-definition cameras or network cameras (IP cameras) deployed at various key locations throughout the park. These cameras typically possess high resolution (e.g., 1080p or 4K), excellent imaging capabilities in low-light conditions, and a wide dynamic range, ensuring clear and accurate video data under diverse environmental conditions. Camera deployment locations include, but are not limited to, park entrances and exits, corridors, parking lots, and public areas to achieve comprehensive park coverage.

[0094] The real-time acquired surveillance video streams are transmitted to the system's data processing module. These video streams form a query set, used for real-time monitoring of behavioral dynamics within the park. The video streams in the query set are continuously acquired, and the system processes them in real time to promptly detect abnormal behavior.

[0095] Meanwhile, the system retrieves video samples of several known categories from a pre-built abnormal behavior database, which constitute the support set. The abnormal behavior database is a pre-built library of video samples storing various labeled abnormal behavior video samples. These samples cover various types of abnormal behavior that may occur within the park, such as fighting, falling, and illegal intrusion. Each abnormal behavior category contains a certain number of video samples, which have been carefully selected and labeled to provide rich feature information for the model.

[0096] While the number of video samples in the support set is limited, they are carefully designed to represent the typical characteristics of each anomalous behavior category. These samples are used during training to build prior knowledge for the model, enabling the system to quickly identify new anomalous behaviors with limited samples. The construction of the support set needs to consider the diversity and representativeness of the samples to ensure that the model maintains high recognition accuracy when faced with different scenarios and behavioral variations.

[0097] By combining real-time collected query sets with pre-defined support sets, the system can efficiently monitor abnormal behavior within the park even in scenarios with few samples. The query set provides real-time behavioral data, while the support set provides feature references for known abnormal behaviors. The combination of the two provides a solid foundation for subsequent feature extraction, fusion, and matching.

[0098] Step 102: Using a pre-trained feature extraction network, process the video frame sequences in the query set and the support set respectively, and extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0099] In step 102, this process is a key step in achieving real-time monitoring with a small sample size, ensuring that the system can comprehensively capture abnormal behavior information in the video from different dimensions.

[0100] First, the feature extraction network processes each frame of video in the query set and supports, extracting spatial features that represent static details. These spatial features primarily capture static information in each frame, such as the shape, texture, and color of objects. These features provide fundamental visual information for subsequent action recognition, helping the system understand the static scene content in the video frames. Using deep convolutional neural networks (such as ResNet50), the system can efficiently convert each frame into a high-dimensional feature vector, which represents the key visual information in the frame.

[0101] Simultaneously, the system extracts temporal features representing global evolution in parallel. These temporal features focus on dynamic changes within the video frame sequence, capturing the continuity and evolution of actions. By modeling the temporal dimension of the video frame sequence, the system can understand the start, development, and end of actions. This process is typically achieved using self-attention mechanisms or recurrent neural networks (such as LSTM), which effectively capture long-distance dependencies between frames, thus providing global temporal information for action recognition.

[0102] Furthermore, the system extracts motion features that characterize the intensity of inter-frame changes. Motion features focus on the differences between frames, highlighting motion information in the video. By calculating the difference between the feature vectors of adjacent frames, the system can capture the intensity of inter-frame changes, thereby identifying key motion features. This feature is crucial for distinguishing between normal and abnormal behavior, as abnormal behavior is often accompanied by significant motion changes.

[0103] By extracting these three features in parallel, the system can comprehensively understand video content from multiple dimensions, including static, dynamic, and motion intensity. Spatial features provide visual details for each frame, temporal features capture the global evolution of actions, and motion features highlight the intensity of changes between frames. The combination of these three features provides a rich information foundation for subsequent feature fusion and matching, enabling the system to achieve high-precision identification and real-time monitoring of complex abnormal behaviors within a park, even in scenarios with few samples.

[0104] Step 103: Based on the motion intensity represented by the action features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the temporal features to generate comprehensive features.

[0105] First, the system uses motion intensity, as represented by motion features, to weight and enhance spatial features. Motion features reflect the intensity of changes between frames, highlighting motion information in the video, while spatial features represent static details within each frame. By using motion features as weight vectors and performing element-wise multiplication with spatial features, the system can dynamically adjust the weights of spatial features based on motion intensity. This weighted enhancement operation strengthens spatial features in motion regions, thereby highlighting the semantic information of key action areas while suppressing background noise interference.

[0106] Next, the system fuses the enhanced spatial features with temporal features. Temporal features capture the global evolution process in the video frame sequence, providing dynamic information about the actions. The fusion operation generates a comprehensive feature by concatenating or adding the enhanced spatial features and temporal features. This fusion method not only preserves the static details of the spatial features but also incorporates the dynamic evolution information of the temporal features, enabling the comprehensive feature to fully characterize abnormal behaviors in the video from multiple dimensions.

[0107] The generation of comprehensive features provides a richer information foundation for subsequent abnormal behavior identification. Through this fusion strategy, the system can more accurately identify key features of abnormal behavior under complex backgrounds and low sample sizes, thereby improving the accuracy and robustness of abnormal behavior identification. This process not only enhances the representational power of features but also provides a more reliable basis for subsequent feature matching and classification decisions.

[0108] Step 104: Input the comprehensive features of the query set into the pre-trained matching model as the query term, adaptively reweight the comprehensive features of the support set, and determine whether there is abnormal behavior in the real-time monitoring video stream of the park based on the query term and the reweighted comprehensive features of the support set.

[0109] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0110] In step 104, the core task of the system is to analyze the combined features of the query set and supports using a pre-trained matching model to determine whether there is abnormal behavior in the real-time monitoring video stream of the park. This process is a key decision-making step in the entire system. Through dynamic adjustment and precise matching, the system ensures that it can achieve efficient and accurate abnormal behavior identification in scenarios with few samples.

[0111] Specifically, the system first inputs the comprehensive features of the query set into a pre-trained matching model as the query term. The core function of the matching model is to adaptively reweight the comprehensive features of the support set based on the features of the query term. Unlike traditional static matching methods, this adaptive reweighting mechanism can dynamically adjust the weights of the support set features according to the characteristics of the query term, thereby more accurately capturing the similarities and differences between the query term and the support set. In this way, the feature representation of the support set can better adapt to the characteristics of the query term, enhancing the model's adaptability to complex scenarios.

[0112] During the matching process, the matching model calculates the similarity between the query term and the combined features of the reweighted support set. This similarity calculation is typically based on a predefined distance metric, such as Euclidean distance or cosine similarity. By minimizing the loss function between the predicted result and the true label, the matching model learns the optimal feature matching strategy during training, enabling it to efficiently identify abnormal behavior in practical applications. When the similarity calculation result indicates a high match between the query term and a certain support set category, the system will determine that corresponding abnormal behavior exists in the real-time monitoring video stream of the park.

[0113] It is worth noting that both the pre-trained feature extraction network and the matching model are trained based on a meta-learning strategy. The core of this strategy lies in training the model's ability to quickly adapt to new tasks by simulating a few-shot learning task. During training, the model learns how to perform effective feature extraction and matching with only a small number of samples by minimizing the loss function between the predicted result and the true label. This training method enables the system to quickly adjust and achieve high-precision recognition when faced with new abnormal behavior categories, significantly improving the system's generalization ability and robustness in real-time monitoring.

[0114] The method for real-time monitoring of abnormal behavior in a park with few samples provided in this invention utilizes action features representing the intensity of inter-frame changes to weight and enhance spatial features representing static details, and deeply integrates them with temporal features. This effectively suppresses environmental noise interference in complex park backgrounds and highlights the semantic expression of key action regions. Furthermore, a pre-trained matching model uses the comprehensive features of the query set as query terms to adaptively reweight the comprehensive features of the support set. This changes the static measurement method in traditional prototype networks where support set features are fixed once extracted, enabling dynamic adjustment of the support set prototype features based on the characteristics of the current sample to be identified. This establishes a deeper semantic association between the query set and the support set, ultimately significantly improving the system's accuracy in identifying complex abnormal behavior within the park and its robustness in real-time monitoring, even with only a few reference samples.

[0115] Furthermore, after obtaining the video streams of the query set and support set, the system first performs frame sequence processing on these videos. The specific operations are as follows:

[0116] The system converts each video sample in the query set and support set into a frame set, with the video frames in the frame set ordered by time sequence. To ensure the representativeness of features and the integrity of dynamic information, the system employs a specific method to extract a fixed number of frame pairs from each video. Specifically, the system divides the frame set into several equal parts, extracts one frame pair from each part, and the spacing between the frame pairs is a preset value G. For example, if N / 2 frame pairs are extracted from each video sample, then each video sample will ultimately generate a representative frame set of N frames. This extraction method enables uniform sampling in the time dimension, ensuring that key dynamic information in the video is captured.

[0117] In this embodiment, the hyperparameters N and G are set empirically, representing the number of frames used for the task and the frame interval for each video sample, respectively. Typically, N is set to 16 and G to 4. This means that each video sample is divided into 8 parts, and a pair of frames is extracted from each part, with a frame interval of 4 frames. This setting reduces computational load while ensuring the integrity of temporal information.

[0118] Then, the extracted frames undergo data augmentation and resizing. During training, the system randomly flips and crops frames to enhance the model's generalization ability; during inference, centered cropping is used to ensure consistency. All frame images are resized to a uniform size, for example, the image is proportionally resized to have a shorter side of 256 pixels, and then cropped to a size of 224×224 pixels. This preprocessing step ensures that the frame images input to the feature extraction network have a uniform format and quality.

[0119] During frame extraction, the frame extraction positions differ between the training and testing phases. During training, to increase data diversity and model generalization ability, the extraction positions of a pair of frames within each part are randomized. However, during testing or real-world use, to ensure consistency and stability, the frame pairs are extracted at the midpoint of each frame set part. This setup provides greater data variability during training while guaranteeing reproducibility during testing.

[0120] Through the above steps, the system completed the extraction and preprocessing of video frames from the query set and support set. This process provides a high-quality data foundation for subsequent feature extraction and matching, ensuring that the system can efficiently identify abnormal behavior within the park under limited sample conditions.

[0121] After preprocessing the frame sequence, the system extracts features from the preprocessed frame images and organizes these features for subsequent processing. This process is performed sequentially, and the specific operations are as follows:

[0122] (1) Feature Extraction. The system uses a pre-trained feature extraction network to process each frame of the image and extract high-dimensional feature vectors. The feature extraction network is based on a deep learning architecture (such as ResNet50) and can efficiently extract features from each frame. In this embodiment, a ResNet50 network pre-trained on a large-scale image dataset (such as ImageNet) is used as the backbone network. This network extracts features from the frame image through convolutional layers and residual connections, and the output feature dimension is 2048×7×7. This high-dimensional feature vector can characterize the static details in the frame image and provide basic information for subsequent action recognition.

[0123] (2) Feature Representation. For each class c and each sample k in the support set, the extracted frame features are represented as follows: ,in This represents the feature information of the nth frame of the kth sample in class c. The frame features of the query sample are represented as follows: ,in This indicates the feature information of the nth frame of the query sample. These feature vectors will serve as the basis for subsequent feature fusion and matching.

[0124] (3) Feature set organization. The feature set S supporting the samples is represented as follows: S c Let c be the feature set of the supporting samples of class c, denoted as , Let be the features of the k-th sample in class c. This organization allows for efficient management and processing of the supporting sample features for each class, providing a clear structure for subsequent feature fusion and matching.

[0125] Furthermore, after completing feature extraction and feature set organization, the system further processes the extracted features to generate three types of features to represent the dynamic information in the video from different perspectives. The specific operations are as follows:

[0126] (1) Feature type classification. The system classifies the extracted features into three types: spatial features, motion features, and temporal features. These three types of features are used to characterize static details, inter-frame variation intensity, and global evolution process in the video, respectively.

[0127] (2) Spatial Features. Spatial features are directly composed of features from each frame and are used to characterize static details in the video. For each class c and each sample k in the support set, the spatial features are represented as follows: ,in, For the set of spatial features of class c, ; Let be the spatial features of the nth frame of a sample in class c. The spatial features of the query sample are represented as follows: These spatial features will serve as the basis for subsequent feature enhancement and fusion.

[0128] (3) Action Features. Action features are generated by differentiating the feature vectors of frame pairs and are used to characterize the intensity of changes between frames. Specifically, for each feature in the support set... The corresponding action features are represented as , where n and N are both even numbers, and the frame sequence group has a total of 2 frame sequences, each sequence has N / 2 feature vectors; This represents the frame-to-action feature sequence of the k-th sample video in class c; This represents the nth frame pair feature of the video. Each frame pair feature is obtained by concatenating the features of two adjacent frames.

[0129] (4) Temporal features. Temporal features are generated by globally modeling the frame sequence and are used to characterize the global evolution process. Specifically, for each feature in the support set... The corresponding time series features are represented as = { } = {[ ] , [ The frame sequence group has a total of 2 frame sequences, and each sequence has N / 2 feature vectors. This represents the temporal features of the k-th video in class c.

[0130] Correspondingly, see Figure 2 The extraction of spatial features representing static details specifically includes:

[0131] 201. Obtain the frame-level feature vector output by the feature extraction network for each frame in the video frame sequence.

[0132] Obtain the 2048×7×7 dimension base feature map output by the backbone network for the video frame sequence. Since the processing flow for each frame feature map of each sample in both the support and query sets is the same, for ease of description, we denote any frame feature map input to CBAM as E∈R. 2048×7×7 That is, the number of channels C=2048, the height H=7, and the width W=7.

[0133] 202. Extract salient target region information from the frame-level feature vector using the spatial attention unit.

[0134] The feature map E is adaptively recalibrated using a cascaded channel attention module and a spatial attention module. The specific calculation process is as follows:

[0135] First, the feature map E is input into the channel attention module. This module generates channel attention weights by performing global pooling and multilayer perceptron processing on the feature map. Subsequently, this weight is multiplied channel by channel by feature map E to obtain the channel-enhanced intermediate feature map E′, calculated using the following formula: .

[0136] Next, the intermediate feature map E′ is input into the spatial attention module. This module generates spatial attention weights through pooling and convolution operations along the channel axis. ( Subsequently, the weight is multiplied element-wise with the intermediate feature map E′ to obtain the final enhanced feature map. The calculation formula is: .

[0137] in, and These are the channel attention module and spatial attention module of CBAM, respectively. The corresponding channel attention weights and spatial attention weights can be obtained from the input feature map. The weights are multiplied by the original feature map to obtain the feature map after feature enhancement.

[0138] 203. The feature channels of the feature vector after extracting information about the significant target region are recalibrated by the channel attention unit, and the spatial features are output.

[0139] The above step 202 is repeated for each frame in the support set and query set to obtain the enhanced single-frame spatial features. Since the shape of the feature maps remains unchanged before and after processing (still 2048×7×7), the system reorganizes these features according to the original data structure to construct the enhanced feature set:

[0140] Support set feature group: The single-frame feature group of the k-th sample of class c in the enhanced support set is denoted as... ,in, This represents the enhanced features of the nth frame of the sample. Correspondingly, the feature set of a single frame for each category of enhanced support samples is denoted as S. cf′ The entire support sample single-frame feature set is denoted as S. f′ .

[0141] Query set feature groups: Similarly, the enhanced query sample single-frame feature groups are denoted as... ,in, This indicates the enhanced features of the nth frame of the query sample.

[0142] Through the above steps, the system successfully extracted and organized highly recognizable single-frame spatial features (S... f′ and Q f′ This lays the data foundation for subsequent fusion with action features and time sequence features.

[0143] Further, see Figure 3 Extracting motion features that characterize the intensity of inter-frame changes, specifically including:

[0144] 301. Obtain the frame-level feature vector output by the feature extraction network for each frame in the video frame sequence.

[0145] First, a pre-trained feature extraction network (backbone network) is used to obtain frame-level feature vectors for each frame in the video frame sequence. These vectors form the basis for all subsequent feature extractions. Assume the frame-level feature set of the k-th sample in class c of the support set is... The frame-level feature groups of the query sample are .

[0146] 302. Perform differential calculation on the frame-level feature vectors of adjacent frames to obtain the original motion features.

[0147] By calculating the difference between feature vectors of adjacent frames, the system can capture changes between frames, thereby generating feature vectors that represent motion information. This process helps to highlight dynamic changes in the video and provides crucial information for subsequent action recognition.

[0148] For the feature vector of each frame Calculate its feature vector with that of adjacent frames. The difference between them is used to characterize the motion information between adjacent frames.

[0149] 303. Calculate the importance weight of each feature channel in the original motion features through the channel attention unit.

[0150] The purpose of channel attention units is to automatically adjust the importance of each channel in the original motion features, highlighting the channels that contribute more to motion information, thereby further enhancing the representational power of the features.

[0151] For each original motion feature Δs n ck The channel attention unit first performs a global average pooling operation to generate channel descriptors C∈R. C .

[0152] The importance weight A is generated by calculating the importance weight of each channel using a multilayer perceptron (MLP) and activation functions such as ReLU and Sigmoid. c ∈R C .

[0153] The generated importance weight A c This indicates the importance weight of each channel, which is used for subsequent feature selection.

[0154] 304. The importance weights are mapped to a preset value range using an activation function, and the original motion features are filtered using the mapped weights to suppress background noise interference and retain key motion information. The filtered features are then used as the action features.

[0155] By using an activation function (such as Sigmoid), the importance weights calculated by the channel attention unit are mapped to a preset numerical range (such as 0 to 1), ensuring that the weight values ​​are within a reasonable range, which facilitates subsequent feature selection operations.

[0156] The mapped importance weight A c Applied to the original motion feature Δs n ck By performing channel-wise multiplication, features are filtered to suppress background noise interference and retain key motion information.

[0157] Δs n ck′ =Δs n ck ⊙A c

[0158] Where ⊙ represents the channel-by-channel multiplication operation, Δs n ck′ These are motion features that have been filtered through channel attention.

[0159] Spatial pooling is performed on the filtered features to reduce the dimension of the feature vector from d′×7×7 to d′. A 2D convolutional layer with a 1×1 kernel is then used to increase the dimension of the feature vector from d′ to d. Finally, a sigmoid layer maps each element of the feature vector to a value between 0 and 1, yielding the final action features.

[0160] Action features Δs after channel attention filtering and spatial pooling n ck′ As the final action feature, it can more accurately represent the dynamic changes in the video and provide a high-quality feature foundation for subsequent action recognition.

[0161] Further, see Figure 4 The extraction of temporal features representing global evolution specifically includes:

[0162] 401. Obtain the frame-level feature vector output by the feature extraction network for each frame in the video frame sequence, and construct a feature sequence set by arranging the frame-level feature vectors output by each frame in the video frame sequence in chronological order.

[0163] The system extracts frame-level feature vectors for each frame in the video frame sequence from a feature extraction network. These feature vectors, extracted using a pre-trained deep learning model (such as ResNet50), characterize the static details of each frame. To capture the dynamic evolution within the video, the system constructs a feature sequence set from these frame-level feature vectors in chronological order. Specifically, for each class c and each sample k in the support set, the system extracts the feature vectors s1 from all frames. ck ,s2 ck ,…,s N ck Arranged in chronological order, forming a set of characteristic sequences S. k cg ={s1 ckg ,s2 ckg ,…,s N ckg Similarly, the set of feature sequences for the query sample is represented as Q. g ={q1 g ,q2 g ,…,q N g This process lays the foundation for subsequent temporal feature extraction.

[0164] 402. The self-attention layer is used to perform global context modeling on the feature sequence set, capture long-distance dependencies between frames, and output the temporal features.

[0165] The system performs global context modeling on the feature sequence set through a self-attention layer. The core of the self-attention mechanism lies in its ability to dynamically calculate the relevance weights between each feature vector in the feature sequence, thereby capturing long-distance dependencies between frames. Specifically, the self-attention layer first calculates the dot product similarity between the query, key, and value matrices, and then uses a scaling factor d... k Normalization is then performed. This process is achieved using the following formula:

[0166]

[0167] Here, f(H) represents the output feature of the attention mechanism, i.e., the new feature representation after weighted aggregation by attention weights; Q, K, and V represent the query, key, and value matrices in the feature sequence, respectively. In this way, the self-attention layer can weighted aggregate the value matrix V according to the calculated attention weights, generating temporal features representing global evolution. Finally, the feature sequence set processed by the self-attention layer... and The temporal features of the supporting and query samples are output respectively, providing important temporal dimension information for subsequent action recognition.

[0168] In its implementation, the system employs a multi-layered stacked self-attention module to enhance the model's ability to capture long-range dependencies. Each layer of the self-attention module performs weighted aggregation of the input feature sequence and passes the results to the next layer. This multi-layered stacked design allows the model to progressively learn more complex temporal relationships, thereby more accurately representing the global evolution process in the video. In this way, the system can not only capture local changes between frames but also understand the global evolution trend of these changes throughout the entire video sequence, providing richer contextual information for the identification of abnormal behavior.

[0169] To further enhance the representation capability of temporal features, the system introduces a cross-attention mechanism on top of the self-attention layer. Specifically, the system divides the features of each frame sequence into two sub-sequences, represented as follows: and These two subsequences contain feature vectors from odd-numbered frames and even-numbered frames, respectively, with each subsequence containing N / 2 feature vectors. The system performs global modeling of these two subsequences using a cross-stack approach. First, the feature vectors of each subsequence are input into a self-attention module for self-attention computation to obtain preliminary feature representations. Then, a cross-attention module interacts the feature representations of the two subsequences to further enhance the feature representation capability. The specific formula is as follows:

[0170]

[0171]

[0172] Where AM represents the attention mechanism module, and These represent the feature representations of the two subsequences at layer n; and These represent the feature representations of the two subsequences at the (n+1)th layer, respectively. In layer n, the features of group 1 are the result of a transformation (such as attention mechanism, normalization, etc.). Through this cross-stacking method, the system can better capture global information in the frame sequence, thereby generating more representative temporal features. Finally, the feature sequence set after processing by the cross-attention mechanism... and The temporal features of the supporting samples and query samples are output respectively, providing a high-quality feature foundation for subsequent action recognition.

[0173] After extracting spatial, motion, and temporal features, the system enters the feature fusion stage to generate comprehensive features. This process involves weighting and enhancing spatial features based on the motion intensity represented by motion features, and then fusing the enhanced spatial features with temporal features to generate comprehensive features.

[0174] See Figure 5 The steps for generating comprehensive features include:

[0175] 501. Using the action features as weight vectors, perform element-wise multiplication with the spatial features to obtain the action-enhanced spatial features.

[0176] The system receives a pre-processed sequence of video frames, which have undergone resizing and data augmentation to ensure that each frame has a uniform format and quality.

[0177] Each frame of the image is processed using a pre-trained feature extraction network (such as ResNet50) to extract high-dimensional feature vectors. The feature extraction network extracts features from the frame image through convolutional layers and residual connections, and the output feature dimension is 2048×7×7. These feature vectors can characterize the static details in the frame image and provide basic information for subsequent action recognition.

[0178] The system fuses single-frame features with frame sequence features. Since these two types of features are one-to-one in terms of quantity (both N frames) and dimension, and each contains rich static and dynamic semantic information, this embodiment uses an element-wise addition method for initial fusion. The calculation formula for the nth frame in the sequence is as follows:

[0179]

[0180] This operation combines the static appearance information of each frame with its temporal context information throughout the video sequence to generate an intermediate feature sequence containing spatiotemporal information. = .

[0181] 502. Perform residual connection between the enhanced spatial features and the unprocessed original spatial features.

[0182] Considering that video sampling is usually performed in frame pairs, and the content and semantics of adjacent frames are highly correlated, this embodiment further refines the intermediate feature sequence S. fg′ Aggregation is performed on a frame-pair basis. Specifically, the feature vectors of two adjacent frames (e.g., frame (n-1) and frame n, where n must be even) are concatenated. Then, a fully connected layer is used to adjust the concatenated high-dimensional vector back to its original dimensions, generating a more compact frame-pair representation. The calculation formula is as follows:

[0183] By performing this operation on the entire sequence, the system can obtain a new intermediate feature set. Its length is half that of the original sequence.

[0184] 503. The feature vector after residual connection is concatenated or added to the time series features to obtain the comprehensive features.

[0185] The system will aggregate the intermediate feature groups Compared with the frame pair features (i.e., action features) extracted in the previous step. The final fusion is then performed. At this point, the two pairs correspond one-to-one in both quantity (N / 2) and dimension. This is due to the frame pair features. It accurately reflects the intensity of the action and can be used as a weight to adjust the intermediate feature set. The significance of the summation is determined. This embodiment uses element-wise dot product for weighting and introduces residual connections to preserve the original information. The calculation formula is as follows:

[0186] in, : Represents the fused features obtained from the nth frame pair, which is usually a feature vector obtained by integrating single-frame features and temporal features;

[0187] : Represents the motion feature weights corresponding to the nth frame pair, which are the weighted results of inter-frame motion intensity after channel attention filtering. Typically, it is a weighted average of the motion features of the nth frame pair. A weight vector with consistent dimensions;

[0188] : Represents the comprehensive features of the nth frame pair, used for subsequent classification or matching.

[0189] Through the above three steps, the system can finally obtain a comprehensive feature set that integrates all information. This high-quality feature representation will be fed into the subsequent Transformer matching model, providing a solid foundation for achieving accurate few-shot recognition.

[0190] After completing feature fusion and generating comprehensive features, the system will enter the feature matching stage. A pre-trained matching model will be used to process the comprehensive features of the query set and support set to identify abnormal behavior in the video. See also Figure 6 The specific steps are as follows:

[0191] 601. Using the pre-trained matching model, the generated comprehensive features corresponding to the support set are linearly mapped to key vectors and value vectors, and the generated comprehensive features corresponding to the query set are linearly mapped to query vectors.

[0192] The system receives the combined features of the generated query set and support set, and uses a pre-trained matching model to map the input feature vector into a key vector K, a value vector V, and a query vector Q.

[0193] The support set feature set is denoted as S′={S 1′ ,…,S C′ The feature set of the queried samples is denoted as Q′={q1′,…,q}. M ′}, where each sample contains M=N / 2 feature vectors, each with dimension d.

[0194] First, the three linear layers (weight matrix W) in the pre-trained model are matched. q W k W v ∈R dk×d Map the supporting features and query features to the key, value, and intermediate query vectors required by the Transformer, respectively:

[0195] key m ck =W k q m ′,value m ck =Wv q m ′,query m =W q q m ′

[0196] Support samples to be indexed as "class-sample-frame" as key n ck value n ck The query sample corresponds to the query. m The dimension is unified as d k =1152.

[0197] 602. Calculate the dot product similarity between the query vector and the key vector, and perform normalization processing to generate an attention weight map.

[0198] For each pair (query) m ,key n ck ) Calculate the scaled dot product similarity and normalize it by layer (denoted as L( ) ):

[0199]

[0200]

[0201]

[0202] in, To query sample features The m-th feature in ';

[0203] yes The kth sample of class c The nth feature;

[0204] , , is the weight matrix used to generate intermediate feature vectors query, key, and value; L is the standard layer normalization.

[0205] It is the scaling factor, numerically equal to The calculated length of the feature is divided by This is because it can alleviate, to some extent, the situation where the result obtained by the dot product has too large a dimension, causing the result to be in a region where the gradient of the softmax function is very small;

[0206] This represents the feature of the m-th query sample and the feature of the k-th sample in the c-th class of the support set. In the attention weights of each feature, among which It is an integer variable with a range of [1, M], representing The corresponding sample feature number;

[0207] Indicates the normalized result Attention weights;

[0208] It is the feature of the k-th sample of class c, obtained after adaptive reweighting, in the m-th frame.

[0209] To ensure dimensionality consistency and facilitate subsequent calculations, the query sample features are also processed by the weight matrix Wv corresponding to the value in the transformer:

[0210]

[0211] U= .

[0212] 603. The value vector is weighted and aggregated using the attention weight map to obtain the reweighted support set comprehensive features.

[0213] transformer calculation results The calculated supporting video sample features are as follows: The features of the queried video samples are Therefore, the feature set of the c-th class of supporting video samples is represented as follows: .

[0214] The class prototype feature is obtained by averaging along the first dimension, and the calculation formula is as follows:

[0215]

[0216] Class c supports the representation of the prototype set as follows: .

[0217] Therefore, the final result is the reweighted comprehensive feature of the support set—the prototype feature of the support sample class. The features of the queried video samples are Among them, u m =W v q m ′ Shared with key-value mapping W v .

[0218] After feature extraction, enhancement, fusion, and adaptive reweighting, the system enters the final abnormal behavior judgment stage. This stage uses the query terms and the reweighted support set to synthesize features and determine the category of abnormal behavior in the real-time monitoring video stream of the park. See also... Figure 7 The specific steps are as follows:

[0219] 701. For each action category in the support set, determine the action prototype of that category based on the comprehensive features of the reweighted support set corresponding to all video samples under that category.

[0220] First, define the prototype set of supporting sample features T = {T} that participate in the computation. 1 ,…,T C}∈R C×M×dk The prototype T of each category c c ={t1 c ,…,t M c}∈R M×dk This represents the combined features of the support set after reweighting for all video samples in this category.

[0221] The prototype of the query sample features is U={u1,…,u M}∈R M×dk .

[0222] 702. Calculate the distance between the query item and the action prototype of each action category to obtain the class distance of the query item for each action category.

[0223] For each pair of class prototypes and query sample features, first calculate the distance between the corresponding feature vectors, then average all the calculated distances to obtain the distance from the corresponding class prototype to the query sample. The calculation formula is:

[0224]

[0225] in, Let m be the feature vector of the c-th prototype. To query the m-th feature vector of a sample, express and The L2 distance between them. For the c-th class of support samples to query samples The distance is calculated by performing the same operation on each pair of class prototypes and query sample features to obtain the final class distance set. .

[0226] 703. Select the action category corresponding to the smallest class distance as the abnormal behavior category of the real-time monitoring video stream of the park.

[0227] The action category corresponding to the smallest class distance is selected as the abnormal behavior category of the real-time monitoring video stream of the park. That is:

[0228]

[0229] That is, take the class distance group The class corresponding to the minimum class distance in the middle.

[0230] For a better understanding of the embodiments of the present invention, please refer to [link / reference]. Figure 8 The flowchart of the method for real-time monitoring of abnormal behavior in a park with few samples according to an embodiment of the present invention is shown.

[0231] Specifically, it includes the following steps:

[0232] S1. Extract multiple frame pairs from the video.

[0233] The system first extracts multiple frame pairs from the park's real-time surveillance video. These frame pairs will be used to capture motion information and behavioral changes in the video. Frame pair extraction can be based on the criticality of the video content, for example, selecting frames that contain significant changes in motion.

[0234] When selecting frame pairs, the system may consider factors such as the time interval between frames and the degree of change in frame content to ensure that the extracted frame pairs can effectively represent the behavior in the video.

[0235] S2, Data Preprocessing.

[0236] Data augmentation is performed on the extracted frame pairs, such as random flipping, rotation, and scaling, to increase data diversity and improve the model's generalization ability.

[0237] Adjust all frame pairs to a uniform size to ensure consistency in subsequent processing. This step typically involves cropping or scaling the frames to a fixed size, such as 224×224 pixels.

[0238] The pixel values ​​of the frame pairs are normalized, for example, by scaling the pixel values ​​to the range of [0, 1] to facilitate model processing.

[0239] S3. Extract features from each sample using a feature extraction network.

[0240] Each frame pair is processed using a pre-trained feature extraction network (such as a convolutional neural network, CNN) to extract high-dimensional feature vectors. These feature vectors can characterize the key information in the frame image.

[0241] For each class c and each sample k in the support set, the extracted frame features are represented as S. k c ={s1ck ,s2 ck ,…,s N ck}, where s N ck This represents the feature information of the k-th sample in class c in the n-th frame.

[0242] S4. Feature sequence recombination to construct single frame sets, frame pair sets, and frame sequence sets.

[0243] The extracted features are reorganized into three types: single-frame feature sets, frame-pair feature sets, and frame sequence feature sets. These three feature sets are used to characterize static details, inter-frame variation intensity, and global evolution processes in the video, respectively.

[0244] Among them, the single-frame feature set is a spatial feature enhanced by generating features from a single frame, used to characterize static details in the video.

[0245] The frame-pair feature set generates motion features from frame pairs, representing motion information in the video.

[0246] The frame sequence feature set generates temporal features from the frame sequence, which characterize the global evolution process in the video.

[0247] S5. Image features enhanced by generating features from a single frame.

[0248] Image features can be enhanced using single-frame features through specific image processing techniques or deep learning models. This might include using convolutional neural networks (CNNs) to further extract features from the image content, or applying advanced image processing algorithms to highlight key information in the image.

[0249] The generated features are used to characterize static details in the video, and these features will be used in subsequent feature fusion processes to provide spatial information.

[0250] S6. Generate motion intensity features through frame pairs.

[0251] Motion intensity features are generated by calculating the feature differences between adjacent frame pairs. This involves calculating the differences between the feature vectors in the frame pairs to capture motion changes in the video.

[0252] Action feature vector These features reflect the intensity of changes in motion between frame pairs and will be used in subsequent steps to provide motion information.

[0253] S7. Generate temporal features from frame sequences.

[0254] Using frame sequences to generate temporal features may involve using recurrent neural networks (RNNs) or Transformer models to understand the temporal relationships and dynamic evolution between frames.

[0255] Temporal feature vector Characterizing the global evolution process in a video is crucial for understanding the temporal structure of actions within the video.

[0256] S8. Perform feature fusion based on properties.

[0257] The image features, motion features, and temporal features generated in steps S5 to S7 are fused. This step involves integrating different types of features into a unified feature representation in order to more comprehensively represent the information in the video.

[0258] Feature fusion can be achieved in various ways, such as concatenation, weighted summation, or using deep learning models to automatically learn the relationships between features. The fused features will be used in subsequent abnormal behavior recognition steps.

[0259] S9. Use Transformer to adaptively match features and construct class prototypes.

[0260] Using a pre-trained Transformer model, the composite features of the query set are matched with the reweighted composite features of the support set. This step involves mapping the query features and support features to a common feature space to facilitate the calculation of their similarity.

[0261] For each action category in the support set, the action prototype for that category is determined based on the combined features of the reweighted support set corresponding to all video samples within that category. The action prototype is obtained by averaging the features of all samples within the category, representing the typical characteristics of that category.

[0262] S10. Calculate the distance between the prototype feature and the query feature, and output the final prediction.

[0263] Calculate the distance between the query term and the action prototype for each action category. This typically involves calculating L2 distance or cosine similarity to quantify the similarity between the query feature and the prototype for each category.

[0264] The action category corresponding to the smallest class distance is selected as the abnormal behavior category of the park's real-time monitoring video stream. This step essentially matches the query features with the most similar category prototype.

[0265] S11. Calculate the loss and train and optimize the model based on the loss.

[0266] To enable the aforementioned feature extraction network and matching model to quickly adapt to new categories under limited sample conditions, this embodiment employs a meta-learning strategy for pre-training. The specific training process is as follows:

[0267] (1) Construct training task (Episode).

[0268] The training data is organized into a series of meta-tasks (Episodes). In each Episode, N action categories are randomly selected from the training set, K samples are selected from each category as the support set, and Q samples are selected from these N categories as the query set. This data organization method simulates the low-sample testing environment in real-world monitoring scenarios.

[0269] (2) Forward propagation and loss calculation.

[0270] For each episode, the constructed support set and query set are input into the model, and the process proceeds sequentially through steps S3 to S10:

[0271] Spatial, action, and temporal features are extracted and fused in parallel using a feature extraction network (S3-S8).

[0272] Adaptive reweighting via a matching model (S9).

[0273] Calculate the distance between the query set and the prototype support set, and obtain the predicted probability distribution (S10).

[0274] Subsequently, the cross-entropy loss function between the predicted results and the true category labels is calculated:

[0275]

[0276] in, The predicted categories generated by the model from the training task. The distance between the query sample and the prototype of class c is used. This loss is then used to train the model, thereby optimizing the model parameters.

[0277] (3) Backpropagation and parameter update.

[0278] Based on the calculated loss value LL, the gradient of the loss function with respect to all trainable parameters of the model (including backbone network weights, attention module weights, and matching model linear layer weights) is calculated using the backpropagation algorithm. An optimizer (such as Adam or SGD) is then used to update the parameters to minimize the loss function.

[0279] After completing the above steps, the system can effectively extract features from videos, generate comprehensive features, construct class prototypes, and ultimately identify abnormal behavior. The entire process forms a closed loop, from data preprocessing to feature extraction, feature matching, and model optimization, ensuring the accuracy and robustness of abnormal behavior detection.

[0280] In addition, embodiments of the present invention utilize image features, motion features, and temporal features of video for action recognition, fully extracting different information from the image frame sequence of a single modality, and fusing them according to the nature of the features, so that the information contained in various features becomes complementary, thereby making the generated action recognition results more in line with expectations.

[0281] Furthermore, this embodiment of the invention uses a transformer method to match each query feature with support set features, constructing support samples that are adaptive to the query features. This method helps to highlight features in the support samples that are more important to the query samples, and the reconstructed support features are more closely matched with the query features and easier to compare, thus improving the accuracy of the results. It can be used in scenarios where the accuracy of action recognition is relatively high.

[0282] The following describes the real-time monitoring system for anomalous behavior in a park with few samples provided by the embodiments of the present invention. The real-time monitoring system for anomalous behavior in a park with few samples described below can be referred to in correspondence with the real-time monitoring method for anomalous behavior in a park with few samples described above.

[0283] This invention provides a real-time monitoring system for anomalous behavior in a park with a small sample size. See [link / reference]. Figure 9 ,include:

[0284] The data acquisition module 910 is used to acquire real-time monitoring video streams of the park as a query set, and retrieve several video samples of known categories from a pre-set abnormal behavior database as a support set.

[0285] The feature extraction module 920 is used to process the video frame sequences in the query set and the support set respectively through a pre-trained feature extraction network, and extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0286] The feature fusion module 930 is used to perform weighted enhancement on the spatial features based on the motion intensity represented by the action features, and to fuse the enhanced spatial features with the temporal features to generate comprehensive features;

[0287] The anomaly detection module 940 is used to input the comprehensive features of the query set into a pre-trained matching model as a query item, adaptively reweight the comprehensive features of the support set, and determine whether there is abnormal behavior in the real-time monitoring video stream of the park based on the query item and the reweighted comprehensive features of the support set.

[0288] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0289] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions from the memory 1030 to execute a method for real-time monitoring of a small number of abnormal behaviors in a campus, including:

[0290] The system obtains real-time monitoring video streams from the park as a query set and retrieves video samples of several known categories from a pre-set abnormal behavior database as a support set.

[0291] The video frame sequences in the query set and the support set are processed by a pre-trained feature extraction network to extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0292] Based on the motion intensity represented by the action features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the temporal features to generate comprehensive features;

[0293] The comprehensive features of the query set are input into the pre-trained matching model as the query term. The comprehensive features of the support set are adaptively reweighted. Based on the query term and the reweighted comprehensive features of the support set, it is determined whether there is abnormal behavior in the real-time monitoring video stream of the park.

[0294] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0295] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0296] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the campus abnormal behavior few-sample real-time monitoring method provided by the above methods, including:

[0297] The system obtains real-time monitoring video streams from the park as a query set and retrieves video samples of several known categories from a pre-set abnormal behavior database as a support set.

[0298] The video frame sequences in the query set and the support set are processed by a pre-trained feature extraction network to extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0299] Based on the motion intensity represented by the action features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the temporal features to generate comprehensive features;

[0300] The comprehensive features of the query set are input into the pre-trained matching model as the query term. The comprehensive features of the support set are adaptively reweighted. Based on the query term and the reweighted comprehensive features of the support set, it is determined whether there is abnormal behavior in the real-time monitoring video stream of the park.

[0301] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0302] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for real-time monitoring of anomalous behavior in a campus with few samples provided by the methods described above, including:

[0303] The system obtains real-time monitoring video streams from the park as a query set and retrieves video samples of several known categories from a pre-set abnormal behavior database as a support set.

[0304] The video frame sequences in the query set and the support set are processed by a pre-trained feature extraction network to extract spatial features representing static details, temporal features representing global evolution, and action features representing the intensity of inter-frame changes in parallel.

[0305] Based on the motion intensity represented by the action features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the temporal features to generate comprehensive features;

[0306] The comprehensive features of the query set are input into the pre-trained matching model as the query term. The comprehensive features of the support set are adaptively reweighted. Based on the query term and the reweighted comprehensive features of the support set, it is determined whether there is abnormal behavior in the real-time monitoring video stream of the park.

[0307] The pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the predicted result and the true label.

[0308] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0309] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0310] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time monitoring of abnormal behavior in a park with few samples, characterized in that, The method comprises: acquiring a real-time monitoring video stream of a park as a query set and calling a plurality of video samples of known categories from a preset abnormal behavior database as a support set; processing video frame sequences in the query set and the support set respectively through a pre-trained feature extraction network, and extracting spatial features representing static details, time sequence features representing global evolution, and motion features representing inter-frame change intensity in parallel; based on the motion intensity represented by the motion features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the time sequence features to generate comprehensive features; inputting the comprehensive features of the query set into a pre-trained matching model as a query item, adaptively re-weighting the comprehensive features of the support set, and judging whether there is an abnormal behavior in the real-time monitoring video stream of the park based on the query item and the re-weighted support set comprehensive features; wherein the pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing the loss function between the prediction result and the true label; the feature extraction network comprises a channel attention unit; extracting motion features representing inter-frame change intensity, specifically comprising: obtaining frame-level feature vectors output by the feature extraction network for each frame in the video frame sequence; differentially calculating the frame-level feature vectors of adjacent frames to obtain original motion features; calculating the importance weights of each feature channel in the original motion features through the channel attention unit; mapping the importance weights to a preset numerical interval through an activation function, and using the mapped weights to filter the original motion features to suppress background noise interference and retain key motion information, and using the filtered features as the motion features; the feature extraction network comprises a spatial attention unit and a channel attention unit; extracting spatial features representing static details, specifically comprising: obtaining frame-level feature vectors output by the feature extraction network for each frame in the video frame sequence; extracting significant target region information in the frame-level feature vectors through the spatial attention unit; recalibrating the feature channels of the feature vectors after extracting the significant target region information through the channel attention unit, and outputting the spatial features; the feature extraction network comprises a plurality of stacked self-attention layers; extracting time sequence features representing global evolution, specifically comprising: obtaining frame-level feature vectors output by the feature extraction network for each frame in the video frame sequence, and constructing the frame-level feature vectors output for each frame in the video frame sequence into a feature sequence set in time sequence; globally modeling the context of the feature sequence set through the self-attention layer to capture long-distance dependency relationships between frames, and outputting the time sequence features; based on the motion intensity represented by the motion features, the spatial features are weighted and enhanced, and the enhanced spatial features are fused with the time sequence features to generate comprehensive features, specifically comprising: using the motion features as a weight vector to element-wise multiply the spatial features to obtain motion-enhanced spatial features; concatenate the enhanced spatial feature with the original spatial feature; concatenate or add the feature vector after residual connection with the time sequence feature to obtain the comprehensive feature.

2. The method of claim 1, wherein, input the comprehensive feature of the query set into a pre-trained matching model as a query item, and perform adaptive re-weighting on the comprehensive feature of the support set, specifically including: linearly map the generated comprehensive feature corresponding to the support set into a key vector and a value vector through the pre-trained matching model, and linearly map the generated comprehensive feature corresponding to the query set into a query vector; calculate the dot product similarity between the query vector and the key vector, and perform normalization processing to generate an attention weight graph; weight and aggregate the value vector through the attention weight graph to obtain the re-weighted support set comprehensive feature.

3. The method of claim 1, wherein, based on the query item and the re-weighted support set comprehensive feature, determine whether there is an abnormal behavior in the park real-time monitoring video stream, specifically including: for each action category in the support set, determine the action prototype of the category based on the re-weighted support set comprehensive feature corresponding to all video samples in the category; calculate the distance between the query item and the action prototype of each action category to obtain the class distance of each action category for the query item; select the action category corresponding to the smallest class distance as the abnormal behavior category of the park real-time monitoring video stream.

4. A park abnormal behavior few-sample real-time monitoring system, characterized in that, including: a data acquisition module configured to acquire a park real-time monitoring video stream as a query set, and retrieve video samples of several known categories from a pre-set abnormal behavior database as a support set; a feature extraction module configured to process video frame sequences in the query set and the support set respectively through a pre-trained feature extraction network, and extract spatial features representing static details, time sequence features representing global evolution, and action features representing inter-frame change intensity in parallel; a feature fusion module configured to weight and enhance the spatial features based on the motion intensity represented by the action features, and fuse the enhanced spatial features with the time sequence features to generate comprehensive features; an abnormality judgment module configured to input the comprehensive features of the query set into a pre-trained matching model as a query item, perform adaptive re-weighting on the comprehensive features of the support set, and determine whether there is an abnormal behavior in the park real-time monitoring video stream based on the query item and the re-weighted support set comprehensive feature; wherein the pre-trained feature extraction network and the pre-trained matching model are trained in advance based on a meta-learning strategy by minimizing a loss function between a prediction result and a true label; the feature extraction network includes a channel attention unit; extracting action features representing inter-frame change intensity, specifically including: obtaining a frame-level feature vector output by the feature extraction network for each frame in the video frame sequence; performing difference calculation on the frame-level feature vectors of adjacent frames to obtain original motion features; calculate the importance weight of each feature channel in the original motion feature through the channel attention unit; mapping the importance weight to a preset numerical interval through an activation function, and screening the original motion feature using the mapped weight to suppress background noise interference and retain key motion information, taking the screened feature as the action feature; the feature extraction network comprises a spatial attention unit and a channel attention unit; the spatial feature representing static details specifically comprises: obtaining a frame-level feature vector output by the feature extraction network for each frame in the video frame sequence; extracting significant target region information in the frame-level feature vector through the spatial attention unit; re-calibrating the feature channels of the feature vector after extracting the significant target region information through the channel attention unit, and outputting the spatial feature; the feature extraction network comprises a plurality of stacked self-attention layers; extracting a time sequence feature representing global evolution, specifically comprising: obtaining a frame-level feature vector output by the feature extraction network for each frame in the video frame sequence, and constructing the frame-level feature vectors output for each frame in the video frame sequence into a feature sequence set in time sequence; global context modeling of the feature sequence set through the self-attention layer to capture long-distance dependency between frames, and output the time sequence feature; based on the motion intensity represented by the action feature, the spatial feature is weighted and enhanced, and the enhanced spatial feature is fused with the time sequence feature to generate a comprehensive feature, specifically comprising: using the action feature as a weight vector, element-wise point multiplication is performed with the spatial feature to obtain the action-enhanced spatial feature; residual connection is performed between the action-enhanced spatial feature and the original spatial feature without processing; the feature vector after residual connection is spliced or added with the time sequence feature to obtain the comprehensive feature.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the park abnormal behavior few sample real-time monitoring method of any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the park abnormal behavior few sample real-time monitoring method of any one of claims 1 to 3.

Images