Lightweight time sequence-based abnormal behavior identification method oriented to edge calculation

Abstract

The invention discloses an edge calculation-oriented lightweight time sequence-based abnormal behavior recognition method, which comprises the following steps of: setting an LSTM (Long Short Term Memory) or GRU (General Resource Unit) module between a feature extractor and a decoder of a MobileInst backbone network, and constructing a recognition model; and training the recognition model, and deploying the trained recognition model on edge equipment to perform abnormal behavior recognition. The problem that an abnormal behavior detection method based on deep learning depends on hardware computing power and is difficult to deploy at edge equipment is solved, and the method is used for edge-based real-time anomaly detection, realizes high-performance anomaly detection in a resource-limited environment and improves the public safety level.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a lightweight temporal anomaly behavior recognition method for edge computing. Background Technology

[0002] The intersection of computer vision and edge computing focuses on enabling intelligent processing and analysis of visual data on resource-constrained edge devices such as mobile terminals, embedded systems, and surveillance cameras. With the widespread adoption of surveillance systems, the amount of video data requiring processing is growing exponentially, making fully automated visual biometric recognition methods a necessity. Analyzing and detecting abnormal activity using these methods can effectively safeguard public safety. Abnormal behavior detection, as a key component, is crucial for timely response to incidents such as accidents, theft, and violence.

[0003] Traditional manual tracking techniques are inefficient and poorly applied in practice. Mainstream deep learning-based methods have made progress in anomaly detection; for example, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are widely used for spatiotemporal anomaly detection. Some studies have used 3D convolutional networks on the UCF-Crime dataset, effectively capturing spatiotemporal features, but the computational cost is high, making them unsuitable for edge devices. Other studies have used EfficientNet-B7 and MobileNet-V2 to build deep learning-based crime detection systems, achieving high accuracy, but facing challenges in resource-constrained environments. Furthermore, methods like the Video Swin Transformer, while effectively modeling long-range dependencies in videos, have high computational costs, making them difficult to apply in real-time and edge deployments. Some hybrid frameworks combining Long Short-Term Memory (LSTM) networks and CNNs have improved anomaly detection capabilities, but they require significant computational resources and have long training and inference times.

[0004] Therefore, most existing spatiotemporal anomaly detection methods rely on hardware computing power and are difficult to deploy on edge devices, which affects their actual integrity and commercial application potential as biometric systems. Summary of the Invention

[0005] The technical problem solved by this invention is to provide a lightweight temporal anomaly behavior recognition method for edge computing, which achieves high-performance spatiotemporal anomaly detection in resource-constrained environments by constructing a novel lightweight framework.

[0006] This invention is achieved through the following technical solution: A lightweight temporal anomaly behavior recognition method for edge computing involves setting up an LSTM or GRU module between the feature extractor and decoder of the MobileInst backbone network to build a recognition model; after training the recognition model, it is deployed on edge devices to perform anomaly behavior recognition. In the recognition model, a feature extractor generates multi-scale spatial local and spatial global features. Then, an LSTM or GRU module performs temporal feature enhancement on the spatial local and spatial global features, outputting a feature X with hidden temporal information. softmax Feature X softmax The outputs are sent to the decoder and the mask decoder, respectively. The mask decoder is a semantically enhanced mask decoder, which modifies the input feature X. softmax First, based on the feature pyramid network, an iterative top-down and bottom-up multi-scale feature fusion strategy is adopted to integrate feature information at different levels. Then, a semantic enhancer is combined to enrich the contextual information of the mask features using global features. At the same time, with the help of the spatial attention mechanism, the key region features related to anomaly detection in the image are selectively enhanced. The decoder is a dual Transformer instance decoder, which processes the input feature X. softmax The local instance decoder refines the query through spatial local features and supplements spatial details; the global instance decoder updates the object query using spatial global features, captures high-level semantics and coarse localization, so that the spatial global features not only have semantic information of the current frame, but also refer to the hidden temporal information in historical frames; the two are combined to generate an accurate instance-level segmentation mask to achieve object-level anomaly recognition. A query passing mechanism and a kernel reuse mechanism are also introduced during the temporal querying and decoding of consecutive frames. The query passing mechanism involves the global instance decoder reusing the target query of consecutive frames to perform cross-spatial and temporal correlation of the input global features and verify the temporal coherence. t The target query is directly used as a frame Initial frame lookup reduces redundant computation and improves timing consistency; The kernel reuse mechanism is that the global instance decoder utilizes the temporal consistency of the segmentation kernels generated by object queries across multiple frames. The segmentation kernels used in the key frame calculation are reused in subsequent frames to obtain hidden temporal information.

[0007] Furthermore, a self-attention layer is introduced based on the feature extractor to model the feature dependency relationship between the spatial and channel dimensions, thereby enhancing the extraction of effective features; The spatial local features extracted by the feature extractor include multi-scale local features X3, X4, and X5, where X3 is a low-level feature containing detailed texture, and X4 and X5 are mid-to-high-level features containing semantic information. The extracted spatial global features are global features X6 containing high-level semantic information, which are used to guide instance decoding; LSTM or GRU modules perform temporal feature enhancement on X3 to X6 respectively, hiding the temporal information into feature X. softmax In the middle, the output features X with hidden temporal information are... softmax The timing enhancement features, after being processed by the LSTM or GRU module, are used for subsequent mask decoding.

[0008] The decoding of the global instance decoder includes: Feature X softmax Enter Object Queries Q and Global Featurers X respectively. G In this context, ObjectQueries Q serves as a learnable instance query vector to decode target instance information from features; GlobalFeaturersX G Global semantic features allow global features to not only contain semantic information of the current frame, but also refer to hidden information in historical frames for anomaly identification. Its output is used as the key and value of the decoder to update the global context of the query vector. Using the Global Instance Decoder as the global Transformer decoder, it receives the Q and X outputs from both. G The output includes an instance query and an instance-level segmentation mask containing global semantics. The decoding of the local instance decoder includes: Feature X softmax Enter Local Features X L Get from X 3 ~ X5's local detail features X L As the key and value of the decoder, it is used to supplement the spatial details of the query vector; the Local Instance Decoder is used as a local Transformer decoder, receiving the output of the global decoder and the local detail features X. L Optimize local details of the query vector; The dual Transformer instance decoder uses output extraction as its output module to generate instance kernels and classification scores from the optimized query vector.

[0009] The semantic enhancement mask decoder performs semantic enhancement through a semantic enhancer (SE). The semantic enhancer (SE) uses the hidden temporal information output by the LSTM or GRU module to strengthen the semantic information of the mask features and improve the segmentation accuracy. The output of the semantically enhanced mask decoder is X.mask It is a mask feature map that has undergone multi-scale feature fusion and SE enhancement, and is used to generate an instance segmentation mask.

[0010] Furthermore, the timing encoding of the LSTM module is as follows: By using gating mechanisms such as input gate, forget gate, and output gate, contextual information across frames can be selectively preserved and updated to capture long-term temporal dependencies; Furthermore, the GRU module uses only update and reset gates to manage the flow of information in the time dimension. After temporal encoding processing by the GRU module, the output is a hidden state containing spatiotemporal fusion features of the entire sequence. The data is input into a classification head located in the global instance decoder, which consists of multiple fully connected layers and activation functions. The fully connected layers achieve feature dimensionality reduction and nonlinear transformation through learnable weight matrices. The activation functions probabilize the output and map the output probability distribution. Finally, the abnormal behavior in the video sequence is determined based on the probability values.

[0011] The query passing mechanism is implemented when the global instance decoder performs a temporal query on the input global features to capture the long-term interaction relationships between different targets in the global scene, and it passes through frames. Generate target query Then, local features are received and interactively generated through cross-attention. t Frame instance segmentation mask; In the When querying a frame, connect the frames. t With frames Query reuse between frames allows the target query of frame t to be passed to frame t. ， Thus, the target query As a frame The initial query, then receiving frames Local feature generation Frame instance segmentation mask.

[0012] The kernel reuse mechanism is that the global instance decoder generates a dynamic convolutional kernel for instance segmentation based on the global features of the input frame. Let the frame... generate Dynamic convolution kernels Each core corresponds to the feature pattern of one instance; In processing the first frame At that time, the frame Generated dynamic convolution kernel K t Passed to frame Using convolution kernels K t Convolution operations are performed on local feature maps to extract hidden states; The feature map of the reuse kernel is then input into the local instance head, and the original score of each pixel belonging to the target is output. Then, the original score is compressed into a probability between 0 and 1 by the sigmoid function to obtain the final segmentation mask.

[0013] Furthermore, the recognition model also employs the following loss function: ; in, , To predict probabilities, Weighting coefficients for specific behavior types. For focusing parameters; , For predicting probabilities Compared with the true value The difference between them; , It represents the total number of possible abnormal behaviors. These are real labels, represented by 0 or 1; These are hyperparameters adjusted based on empirical verification of performance.

[0014] Furthermore, the training of the constructed recognition model includes the following operations: Multi-scale features are extracted from the MobileInst backbone network. Video data frames are processed through the temporal module LSTM or GRU module to generate inter-frame hidden information. Global features and object queries are processed by dual Transformer decoders to generate kernels. The kernels are then passed through the query and reused to the local decoder. The local features are combined to generate preliminary results. The semantic enhancement mask decoder then refines the output to produce a semantically meaningful mask, thus completing the forward inference. Then, the classification loss function Focal Loss L is used. focal Constraint instance classification, Binary Cross-EntropyLoss function L BCE Mask loss is used to optimize pixel matching, and the Smooth L1 Loss function L is used. Smooth L1 Regression loss improves coordinate accuracy, and then the total loss is fused according to weights; Finally, the error is backpropagated and the parameters of the backbone network, Transformer layer, and temporal module are updated sequentially. The process is iterated until the loss converges, thus achieving a closed loop in the training of the recognition model.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects: This invention combines the lightweight MobileInst architecture with the temporal modules MobLSTM / MobGRU to propose a lightweight temporal model suitable for edge computing, and further optimizes performance using a query-passing mechanism. MobLSTM, by integrating a temporal LSTM model, enhances the detection capability for gradual or subtle anomalous activities, and improves the detection capability for anomalous behaviors that develop slowly over time and show little change. This invention solves the problems of existing deep learning-based anomaly detection methods relying on hardware computing power and being difficult to deploy on edge devices, controlling computational load while maintaining accuracy. It achieves high-performance anomaly detection in resource-constrained environments, improving public safety.

[0016] This invention improves the model's architecture design and information flow mode, ensuring that the model can still perform detection tasks stably and efficiently in resource-constrained environments such as edge devices. The MobGRU of this invention can manage information with fewer gating mechanisms, minimizing the computational burden while ensuring a certain level of detection accuracy, making it more suitable for handling dynamic and short-term temporal behaviors.

[0017] The query passing mechanism of this invention reduces computational redundancy and improves temporal continuity by reusing queries obtained in consecutive frames, further optimizing computational efficiency on edge devices and reducing repetitive computation steps to improve the real-time response speed of the system. This enables a high-performance anomaly detection system to play a practical role in resource-constrained scenarios and meet real-world application needs.

[0018] This invention, employing MobLSTM and MobGRU technologies, enables frame-level anomaly detection on the UCF-Crime dataset. Compared to the MobileInst backbone network (79.91% accuracy), the accuracy reaches approximately 92.82% when integrating MobLSTM and 89.59% when integrating MobGRU, significantly improving detection performance. It can effectively identify aberrations such as abuse, arrest, and arson, achieving superior recognition results through technological characteristics and highlighting the advantages of the solution.

[0019] This invention, employing MobLSTM and MobGRU technologies, exhibits significant advantages in frame rate performance and high computational efficiency, making it suitable for edge devices. Compared to the lightweight model MobileNet-V3 (Large) (27.2 FPS), MobLSTM and MobGRU can process 90.8 and 92.3 frames per second, respectively. Furthermore, the model parameters and computational load are reasonably controlled, with MobLSTM having 15 million parameters and MobGRU having 14 million, resulting in approximately 24.25G floating-point operations per second. In actual testing, MobLSTM's frame rate ranged from 37.9 to 65.2 FPS across various edge devices. Even on low-cost devices like Luckfox, it meets practical usage requirements, fully demonstrating the solution's advantages in performance and adaptability.

[0020] Experiments were conducted on the model of this invention using multiple sets of different random seeds. Compared with conventional experiments without controlling the random seed variable, the performance deviation of the model of this invention was effectively controlled. Taking MobLSTM as an example, its F1-Score standard deviation was only ±0.06, and its FPS standard deviation was ±0.12. With its stable experimental design, the performance uncertainty caused by random seed fluctuations was overcome, thus enabling the model of this invention to have the advantage of reliable convergence and verifying the technical value of the scheme in terms of stability. Attached Figure Description

[0021] Figure 1 This is an architecture diagram of MobLSTM and MobGRU, lightweight time-series models based on MobileInst. Figure 2 For frames With frames A diagram illustrating the query transmission mechanism between them; Figure 3 A schematic diagram of the core reuse mechanism; Figure 4 Hyperparameter sensitivity heatmaps for MobLSTM and MobGRU models; Figure 5 The activation status of the MobLSTM forget gate and MobGRU update gate at the video frame timing step is shown, where the color intensity represents the degree of gating activation. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to embodiments. These descriptions are for illustrative purposes only and are not intended to limit the scope of the invention.

[0023] The proposed method for edge computing-oriented lightweight temporal anomaly behavior recognition constructs a recognition model based on the MobileInst backbone architecture and fused with LSTM or GRU. Both lightweight temporal models, MobLSTM and MobGRU, can be used for recognition.

[0024] The present invention provides a lightweight temporal abnormal behavior recognition method for edge computing, which involves setting up an LSTM or GRU module between the feature extractor and decoder of the MobileInst backbone network to build a recognition model; after training the recognition model, it is deployed on edge devices to perform abnormal behavior recognition. In the recognition model, a feature extractor generates multi-scale spatial local and spatial global features. Then, an LSTM or GRU module performs temporal feature enhancement on the spatial local and spatial global features, outputting a feature X with hidden temporal information. softmax Feature X softmax The outputs are sent to the decoder and the mask decoder, respectively. The mask decoder is a semantically enhanced mask decoder, which modifies the input feature X. softmax First, based on the feature pyramid network, an iterative top-down and bottom-up multi-scale feature fusion strategy is adopted to integrate feature information at different levels. Then, a semantic enhancer is combined to enrich the contextual information of the mask features using global features. At the same time, with the help of the spatial attention mechanism, the key region features related to anomaly detection in the image are selectively enhanced. The decoder is a dual Transformer instance decoder, which processes the input feature X. softmax The local instance decoder refines the query through spatial local features and supplements spatial details; the global instance decoder updates the object query using spatial global features, captures high-level semantics and coarse localization, so that the spatial global features not only have semantic information of the current frame, but also refer to the hidden temporal information in historical frames; the two are combined to generate an accurate instance-level segmentation mask to achieve object-level anomaly recognition. A query passing mechanism and a kernel reuse mechanism are also introduced during the temporal querying and decoding of consecutive frames. The query passing mechanism involves the global instance decoder reusing the target query of consecutive frames to perform cross-spatial and temporal correlation of the input global features and verify the temporal coherence. t The target query is directly used as a frame Initial frame lookup reduces redundant computation and improves timing consistency; The kernel reuse mechanism is that the global instance decoder utilizes the temporal consistency of the segmentation kernels generated by object queries across multiple frames. The segmentation kernels used in the key frame calculation are reused in subsequent frames to obtain hidden temporal information.

[0025] Furthermore, a self-attention layer is introduced based on the feature extractor to model the feature dependency relationship between the spatial and channel dimensions, thereby enhancing the extraction of effective features; The query passing mechanism is implemented when the global instance decoder performs a temporal query on the input global features to capture the long-term interaction relationships between different targets in the global scene, and it passes through frames. Generate target query Then, local features are received and interactively generated through cross-attention. t Frame instance segmentation mask; In the When querying a frame, connect the frames. t With frames Query reuse between frames allows the target query of frame t to be passed to frame t. ， Thus, the target query As a frame The initial query, then receiving frames Local feature generation Frame instance segmentation mask.

[0026] The kernel reuse mechanism is that the global instance decoder generates a dynamic convolutional kernel for instance segmentation based on the global features of the input frame. Let the frame... generate Dynamic convolution kernels Each core corresponds to the feature pattern of one instance; In processing the first frame At that time, the frame Generated dynamic convolution kernel Passed to frame Using convolution kernels Convolution operations are performed on local feature maps to extract hidden states.

[0027] The following is combined with Figure 1 The backbone network and the recognition model are explained separately.

[0028] 1. MobileInst backbone network MobileInst, a lightweight spatial feature extractor, mainly includes: (1) Feature extractor based on Mobile Transformer: This feature extractor is based on Mobile Transformer, and integrates depthwise separable convolution and self-attention mechanism to generate multi-scale spatial local features X3, X4, X5 and global features X6 to capture the detailed texture and high-level semantic information of the image.

[0029] (2) Semantic-enhanced mask decoder: First, based on the feature pyramid network (FPN), an iterative top-down and bottom-up multi-scale feature fusion strategy is adopted to integrate feature information at different levels; then, combined with the semantic enhancer, the contextual information of the mask features is enriched by global features; at the same time, with the help of the spatial attention mechanism, the key regional features related to anomaly detection in the image are selectively enhanced to improve the accuracy of segmentation mask.

[0030] (3) Dual Transformer instance decoder: It consists of a query-based local instance decoder and a global instance decoder. The local instance decoder refines the query and supplements spatial details through local features X3, X4, and X5. The global instance decoder updates the object query using global feature X6, capturing high-level semantics and coarse localization. The outputs of the two (local instance decoder and global instance decoder) are combined to generate an accurate instance-level segmentation mask, realizing object-level anomaly recognition.

[0031] The MobLSTM model combines the lightweight spatial backbone network of MobileInst with the advanced temporal modeling capabilities of LSTM. The MobGRU model combines the MobileInst lightweight spatial backbone network with gated recurrent units (GRUs) in an end-to-end manner, adapting to the dual requirements of computational efficiency and temporal understanding in video analytics scenarios.

[0032] Figure 1 shows the architecture of the lightweight temporal models MobLSTM and MobGRU based on MobileInst, illustrating the complete process from input image to output abnormal behavior detection result. It includes core components such as Mobile Transformer feature extraction module, Dual Transformer Instance Decoder, Semantic-enhanced Mask Decoder, LSTM / GRU temporal module and query passing mechanism, which are used to illustrate how the present invention achieves real-time abnormal behavior detection on edge devices by fusing spatial features and temporal information.

[0033] In the Mobile Transformer feature extraction module in the left vertical column (① ~ ⑥), X3, X4, and X5 are multi-scale local feature maps extracted by Mobile Transformer. X3 is a low-level feature containing detailed texture, while X4 and X5 are mid-to-high-level features containing more semantic information. X6 is a global feature map containing high-level semantic information used to guide instance decoding.

[0034] The LSTM / GRU units perform temporal feature enhancement on X3 to X6 respectively, and output features X with hidden temporal information. softmax (LSTM / GRU units have the ability to save the state of consecutive frames, which can hide timing information in feature X) softmax (In the middle), its timing enhancement features after LSTM / GRU processing are used for subsequent mask decoding.

[0035] Top module For a dual Transformer instance decoder, Object Queries Q is a learnable instance query vector used to decode target instance information from features; Global Featurers X G These are global semantic features from X6, used as keys and values ​​in the decoder to update the global context of the query vector; ⑥X softmax Enter into Global Featurers X G In this way, the global features not only have the semantic information of the current frame, but also refer to the hidden information in the historical frames (the inclusion of hidden temporal information in the features enables the prediction module to more accurately predict the probability of abnormal behavior in the current frame).

[0036] The Global Instance Decoder is a global Transformer decoder used to receive Q and X. G It also outputs instance queries that include global semantics.

[0037] Local Features X L It is from X 3 ~ X5's local detail features, serving as keys and values ​​in the decoder, supplement the spatial details of the query vector. The Local Instance Decoder is a local Transformer decoder that receives the output of the global decoder and Local Features X. L Output X L Optimize local details of the query vector; Output extraction is the output module of the dual Transformer instance decoder, which generates instance kernels and classification scores from the optimized query vector for subsequent mask generation.

[0038] Middle module For semantic enhancement mask decoders, SE (Semantic Enhancer) is the semantic enhancer that uses the hidden information processed by LSTM / GRU to enhance the semantic information of the mask features and improve segmentation accuracy; X maskIt is a mask feature map after multi-scale feature fusion and SE enhancement, used to generate instance segmentation masks.

[0039] Unsample is an upsampling operation used to restore a low-resolution mask feature map to the same size as the input image, ensuring the accuracy of segmentation details.

[0040] LSTM Block (Timing Module) Includes forget gates, input gates, and output gates to remember cell states, used to capture long-term temporal dependencies; GRU Block It includes reset gates and update gates, which improve computational efficiency by simplifying the gating mechanism and are suitable for capturing short-sequence dynamics.

[0041] 2. Implementation of the MobLSTM model (1) Spatial feature extraction: each frame in the video First, the data is fed into the MobileInst backbone network, where the Mobile Transformer feature extractor extracts multi-scale spatial features and encodes them into vector representations. F t ,in F t It integrates spatial information at different scales, including global semantics and local details.

[0042] (2) LSTM temporal coding: The spatial features of consecutive frames are embedded into the LSTM module. The gating mechanism of input gate, forget gate and output gate is used to selectively retain and update the context information across frames in order to capture long-term temporal dependencies. The following formula can be used to capture the context information: ; in These are the forget gate, input gate, and output gate, respectively. This is a cell state used to maintain long-term dependencies across video frames, and to store short-term temporal information for transfer between LSTM units. For MobileInst at time step Extracted features; For candidate memory cell states, activate them using the tanh function, making their values ​​(-1, 1); tanh is the hyperbolic tangent activation function, which maps input values ​​to the (0,1) interval and is used to scale candidate cell states and cell states; б is the sigmoid activation function, which maps input values ​​to the (0,1) interval and is often used in gating mechanisms to represent the degree of opening of the gate; h t yes tThe hidden state of a time step is used to pass information to the next time step, and also serves as the output of the current time step, providing information to other parts of the network; h t-1 yes t The hidden state at time -1 carries information from the previous time step; C t-1 It represents the cell state at the previous moment; b f , b i , b o , b C These are the bias vectors used in the calculation of the forget gate, input gate, output gate, and candidate cell state, respectively, which are learned during the training process. W f , W i , W o , W C These are the weight matrices used in the calculation of the forget gate, input gate, output gate, and candidate cell state, respectively, which are learned and adjusted during training using the backpropagation algorithm.

[0043] (3) Anomaly classification: final hidden state Using the fully connected classification head of the global instance decoder, a linear transformation is applied to the hidden state. Mapping to the same dimension as the number of categories n, and processing using the softmax function, the system judges based on the probability values ​​of different abnormal behaviors, selecting the one with the highest probability value as the anomaly.

[0044] 3. Implementation of the MobGRU model (1) Spatial feature extraction: each frame in the video First, the data is fed into the MobileInst backbone network, where it is processed by the Mobile Transformer feature extractor to generate multi-scale spatial feature embeddings. .

[0045] (2) GRU temporal coding: embedding features The data is input frame by frame into the GRU module. The GRU uses only update and reset gates to manage the flow of information over time, implemented using the following formula: ; in To update the door, control how much of the previous hidden state is retained; To reset the door, determine how much past information to use to integrate with the new information; The current hidden state; σ is the Sigmoid activation function, which maps the linear transformation result to (0,1) to achieve gating on / off control; Wz To update the weight matrix of the gate, used for... Perform a linear transformation to match the dimensions of the input and output; Wr To reset the weight matrix of the gate, for Perform a linear transformation; The hidden state (historical time sequence information) of the previous time step is concatenated with the input of the current frame; Wh It is the weight matrix of the candidate states, used to... Perform a linear transformation; For element-wise multiplication between the two, use a reset gate to filter historical hidden states, r t The closer a value is to 1, the more information in the hidden state is retained; the closer a value is to 0, the less information is retained. bz , br、bh These are the bias vectors for the update gate, reset gate, and candidate state, respectively, used to adjust the linear transformation results and improve the fitting ability; The new state generated by combining historical information filtered by the reset gate with the current input information provides new information for updating the hidden state, including dimensions and... Same; tanh is the hyperbolic tangent activation function, which maps the input to the range (-1,1) and is used to introduce nonlinearity into the candidate state to enhance the expressiveness of the model.

[0046] (3) Anomaly classification: After processing by the GRU temporal coding module, the output contains hidden states containing spatiotemporal fusion features of the entire sequence. The data is input into a classification head (global instance decoder) consisting of multiple fully connected layers and activation functions. The fully connected layers perform feature dimensionality reduction and nonlinear transformation through learnable weight matrices. The activation functions probabilize the output and map the output probability distribution. Finally, the abnormal behavior in the video sequence is determined based on the probability values, thus completing the anomaly prediction.

[0047] 4. Mechanisms to enhance the adaptability of edge devices To address the deployment challenges in resource-constrained environments at the edge, this invention introduces key mechanisms such as query passing, kernel reuse, and adaptive self-attention.

[0048] (1) Query passing mechanism: By reusing queries of consecutive frames, the temporal coherence in feature propagation processing is increased, and the frames are passed through the query passing mechanism. target query Directly as a frame initial query This reduces redundant computation and improves timing consistency. Represented as ,in For the target query of the previous frame, This serves as the initial query for subsequent frames and is passed to the local instance decoder, reducing redundant computation.

[0049] Figure 2 This is a temporal training diagram illustrating the query passing mechanism, demonstrating how efficient temporal modeling is achieved between adjacent frames in a video sequence by reusing the target query. Figure 1 Timing extension of the Dual Transformer InstanceDecoder module; Figure 2 The left-hand module is a frame. t Related components, ①Frame t is the first frame in the video sequence. t The frame serves as the baseline frame for temporal training. ②Global Features X G From frame t The extracted global features contain high-level semantic information to guide the updating of the target query. ③ Object Queries Q is the target query vector output from the global instance decoder of frame t, used for instance segmentation and tracking. ④ The Global Instance Decoder is the frame... t A global decoder used to receive X G And generate the target query Q. Figure 2 Global features X G ②, Target query Q ③ and Global instance decoder ④ and Figure 1 Global Featurers X G Object Queries Q and Global Instance Decoder correspond to respectively; for the first [item] in the video... t Frame t utilizes a self-attention mechanism to model the cross-spatial and temporal relationships of global input features, capturing the long-term interaction relationships between different targets in the global scene, and ultimately generating a target query. .

[0050] The right-hand module is a frame. Related components, ⑤Frame For frames following frame t at an interval of δ in the video sequence, this is used to verify the temporal coherence. ⑥Local Features X L From frame The global features extracted contain detailed information from the frame to optimize the instance segmentation mask. ⑦Object Queries Q' is the frame... Reuse the target query Q of frame t as its initial query. ⑧ Local Instance Decoder for frames The local decoder is used to receive local features and an initial query to generate an instance segmentation mask for the frame. Figure 2 Local instance decoder in ⑧ and Figure 1 This corresponds to the LocalInstance Decoder in the context.

[0051] Query Passing connects frame t and frame t. The reuse between frames passes the target query Q of frame t to frame t. The core mechanism for reducing computation is a local feature transfer body containing query information. Because local details can change significantly between adjacent frames, the Query Passing Path is not directly reused, but rather integrated with the first frame. Frame The new local features interact again through cross attention, dynamically updating detailed information.

[0052] (2) Kernel reuse mechanism: By leveraging the temporal consistency of the segmentation kernels generated through object queries across multiple frames, the kernels used for keyframe computation can be reused in subsequent frames. For example, the segmentation kernels generated from keyframes can be reused. Reusing it in subsequent frames reduces computational overhead; Assuming time Global decoder generation Dynamic convolution kernels In subsequent frames In the middle, for local feature maps reuse ,pass To obtain hidden temporal information, i.e., using convolutional kernels. For local feature maps Convolutional operations are performed to extract hidden temporal information; finally, the feature map of the reuse kernel is input into the local instance head, and the original score of each pixel belonging to the target is output. Then, the original score is compressed into a probability between 0 and 1 by the sigmoid function to obtain the final segmentation mask.

[0053] Figure 3 This demonstrates the core workflow of inter-frame feature processing and mask generation in video instance segmentation, mainly comprising five key components. ① The Global Instance Decoder is a frame-wide instance decoder whose core function is to generate dynamic convolutional kernels for instance segmentation based on the global features of the input frame. (Corresponding to...) Figure 1The first part of the global decoder uses a dual Transformer to generate Kernels; the second part uses dynamic convolutional kernels generated by the global instance decoder, with each kernel corresponding to the feature pattern of an instance. Figure 1 The query passing mechanism in the code is used to pass the kernel generated by frame t to frame t. (In time-series delivery, this avoids redundant calculations and reduces the computational burden on edge devices.) ③Local Features X L It is the interval between frames t. δ The subsequent local feature data, and Figure 1 The MobileInst backbone network and feature generation correspond to the generation of features, containing fine-grained spatial details of the frame, and reusing kernels. The target mask is generated after combining.

[0054] ④ The Local Instance Decoder utilizes local features and reused kernels. Accurately locate and segment instances in the frame by... Applied to Generate hidden features containing instance spatial locations, providing the foundation for the final mask generation. ⑤ The Generated Mask is a local instance decoder. The segmentation mask of each instance in the frame, and Figure 1 The semantic enhancement mask decoder corresponds to the output of the final segmentation mask. Each mask corresponds to an instance, and pixel-level instance differentiation is achieved by marking the pixel region of the instance through binarization. This is one of the final visualization results of video instance segmentation.

[0055] (3) Adaptive self-attention mechanism: The feature extractor based on Mobile Transformer introduces a self-attention layer to model the feature dependency relationship between the spatial and channel dimensions. Its mathematical expression is: Where Q, K, and V are the query matrix, key matrix, and value matrix derived from the input feature map, respectively, and K... T This is the transpose of the key matrix. This is the feature space dimension; through this mechanism, the model can automatically focus on key regions of the image, enhancing the extraction of effective features.

[0056] 5. Loss Function This invention integrates Focal Loss, Smooth L1 Loss, and Binary Cross-Entropy Loss, assigning them different weights as the loss function of the recognition model. The semantic-enhanced mask decoder is used to refine the mask edges, and the output X_mask needs to be compared with the real mask to calculate the pixel-level error, corresponding to the Binary Cross-Entropy Loss.

[0057] The overall mathematical expression for the loss function is shown below: ; in: Focal Loss is mainly used to solve the problems of imbalance between positive and negative samples in training data and large differences in the number of samples of different classes.

[0058] In the UCF-Crime dataset, normal behavior samples far outnumber abnormal behavior samples, and the number of samples for different abnormal behavior types also varies significantly. Focal Loss can dynamically adjust sample weights, allowing the model to focus more on samples that are difficult to classify. The formula is: ,in To predict probabilities, Weighting coefficients for specific behavior types. To focus on the parameters. By introducing an adjustment factor. ,when When the sample size is large, meaning it's easy to classify, the overall weight of this factor is small, so its weight is reduced; when... When a sample is relatively small, indicating a difficult-to-classify sample, the overall weight of this factor is increased, thus making the model more focused on learning difficult-to-classify samples.

[0059] Smooth L1 Loss is used to optimize the accurate temporal localization of anomalous behavior in video sequences. Anomalous behavior typically consists of several consecutive frames with discernible temporal boundaries between them, and Smooth L1 Loss helps the model better perceive these boundaries. Its mathematical expression is: , in For predicting probabilities Compared with the true value The difference between the values ​​can be used to determine the gradient. When the difference is small, the gradient changes smoothly, which helps stabilize training. When the difference is large, it can reduce the impact of outliers and avoid gradient explosion, making it suitable for handling abnormal behavior detection tasks in continuous data.

[0060] BCE Loss is used for binary classification tasks, primarily assisting in achieving a stable "yes" or "no" judgment for each anomalous behavior. It is suitable for predicting various anomaly types. The mathematical expression is: , in It represents the total number of possible abnormal behaviors. These are real labels, represented by 0 or 1; This represents the probability of predicting abnormal behavior. The loss is measured by calculating the cross-entropy between the predicted probability and the true label. The closer the prediction is to the true label, the smaller the loss; conversely, the greater the loss, prompting the model to make a more accurate binary classification.

[0061] These three loss functions work together to complement each other. These are hyperparameters adjusted based on validation performance experience, used to balance the weights of these three losses in the overall loss function to achieve better model training results.

[0062] This invention addresses the balance between accuracy and efficiency in anomaly detection for surveillance videos through a timing module (MobLSTM / MobGRU replaceable features) and a multi-module collaborative architecture. The following section, using specific experiments, focuses on illustrating the impact of the MobLSTM / MobGRU timing module, gating mechanism, hyperparameters, and other features.

[0063] The basic experimental configuration for this embodiment is given below: Hardware: CPU is a 14 vCPU Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz, RAM is 64GB, GPU is a 1*24 GB NVIDIA RTX 3090.

[0064] Software: Python 3.7, PyTorch 1.10, CUDA 11.3, cuDNN 9.7.0.

[0065] Dataset: The UCF-Crime dataset is used, containing over 1900 untrimmed surveillance videos totaling approximately 128 hours, covering various criminal activities such as theft, assault, and robbery. The dataset is divided into a training set of 1500 videos and a test set of 400 videos, representing 80% and 20% of the total dataset, respectively.

[0066] The training of the recognition model is shown below: The model training process mainly includes four parts: data input, forward inference, loss calculation, and backward optimization. First, the training data is input, and multi-scale features are extracted by the MobileInst backbone network. Video data frames are processed by the LSTM / GRU module to generate inter-frame hidden information. Global features and object queries are processed by dual Transformer decoders to generate kernels. The kernels are then passed through the query and reused to the local decoders. The local features are combined to generate preliminary results, which are then refined by the semantic enhancement mask decoder to output a semantically meaningful mask, completing the forward inference. Then, Focal Loss is used to constrain instance classification, Binary Cross-Entropy Loss is used to optimize pixel matching, and Smooth L1 Loss is used to improve coordinate accuracy. Finally, the total loss is fused according to the weights. The error is backpropagated and the parameters of the backbone network, Transformer layer, and temporal module are updated in sequence. The process is iterated until the loss converges, thus achieving a closed loop in model training.

[0067] Design differences between MobLSTM and MobGRU In this invention, MobLSTM and MobGRU are replaceable timing modules, both with a dropout value of 0.5, a batch size of 16, and a hidden dimension of 256. Controlled-variable experiments were conducted to verify the differences between the two modules in terms of accuracy, efficiency, and robustness.

[0068] (1) MobLSTM module Combining MobileInst spatial feature extraction with an LSTM long short-term memory network, a triple gating mechanism (input gate, forget gate, and output gate) enhances the ability to capture progressive, long-term anomalous behaviors (such as burglary and violence). On the UCF-Crime dataset, the frame-level anomaly detection accuracy reaches 92.82%, with a frame rate of 90.8. For long-term anomalous behaviors such as theft and vandalism, the F1-Scores reach 96.93% and 96.37%, respectively, significantly outperforming the 79.71% accuracy of the MobileInst backbone network.

[0069] (2) MobGRU module Using only update and reset gates reduces computational steps, focusing on efficient detection of dynamic, short-term anomalies while balancing speed and accuracy. On the UCF-Crime dataset, it achieves an accuracy of 85.59% and a frame rate of 92.3. For attacks on others and accidents, it achieves F1-Scores of 93.50% and 92.58% respectively, making it suitable for detecting latency-sensitive scenarios.

[0070] Figure 4This is a heatmap of hyperparameter sensitivity for the MobLSTM and MobGRU models, used to analyze the impact of hyperparameters λ_cls (classification loss weights) and λ_reg (regression loss weights) on the model's F1 score. The horizontal axis represents λ_reg, and the vertical axis represents λ_cls; the color intensity corresponds to the F1 score. The results show that both models achieve the highest F1 scores when λ_cls and λ_reg are between 0.3 and 0.4. Beyond this range, the model performance gradually declines, indicating the need to balance classification and regression losses. Furthermore, the graph shows that the MobLSTM model consistently achieves a higher F1 score than the MobGRU model; therefore, under the same parameter conditions, the MobLSTM model performs better.

[0071] Figure 5 The activation status of the MobLSTM forget gate and MobGRU update gate at video frame time steps is displayed, with colors representing the degree of gating activation. Based on the LSTM / GRU gating mechanism, the forget gate controls the retention of historical information, while the update gate balances the new frame with the historical state. Anomaly detection on the UCF-Crime dataset reveals that the MobLSTM forget gate, with high adaptation values, handles long-term dependency anomalies, while with low adaptation values, it handles short-term sudden anomalies. The MobGRU update gate, with low adaptation values, handles stable behavior patterns, while with high adaptation values, it handles abrupt behavior changes. Both gates have advantages in complex long-term behavior and scenarios with high real-time requirements. The visualization results provide a basis for analyzing the model's identification of various anomalies.

[0072] The query passing mechanism passes frames by reusing query vectors from consecutive frames. Query Directly as a frame initial query This reduces redundant computation to improve timing consistency. Experiments show that this mechanism keeps the frame rate of MobLSTM between 37.9 and 65.2 on edge devices.

[0073] The kernel reuse mechanism utilizes the segmentation kernel generated from keyframes. Multiplexing in subsequent frames reduces computational overhead; the specific mathematical expression is as follows: Ablation experiments show that using this mechanism, the model's floating-point operations (FLOPs) can be controlled to around 24.25G, and the parameter sizes of MobLSTM and MobGRU are 15 million and 14 million respectively, making them suitable for memory-constrained edge devices. The adaptive attention mechanism can dynamically adjust the attention given to features, enhancing the detection capability for small targets or occluded scenes.

[0074] Due to its technical architecture adapted to resource-constrained scenarios, this invention can be deployed in edge environments compared to traditional solutions that rely on high-performance hardware. Taking Rockchip's Orange Pi development board as an example, the trained model's Python code is first converted into an ONNX model using RKNN-ToolKit2. Then, the ONNX model is deployed to the development board using ADB tools. Other edge devices can then follow the same process using their corresponding tools. This convenient and readily available real-time biometric system overcomes the problems of insufficient resources in edge scenarios and poor scalability of traditional monitoring solutions. Tested and verified in various monitoring scenarios such as streets, shops, schools, and parking lots, it effectively provides a scalable solution for real-world monitoring and has significant value in improving public safety.

[0075] The embodiments given above are preferred examples for implementing the present invention, and the present invention is not limited to the above embodiments. Any non-essential additions or substitutions made by those skilled in the art based on the technical features of the present invention are within the protection scope of the present invention.

Claims

1. A lightweight temporal anomaly behavior recognition method for edge computing, characterized in that, The process involves setting up an LSTM or GRU module between the feature extractor and decoder in the MobileInst backbone network to build a recognition model; after training the recognition model, it is deployed on edge devices to identify abnormal behavior. In the recognition model, a feature extractor generates multi-scale spatial local and spatial global features. Then, an LSTM or GRU module performs temporal feature enhancement on the spatial local and spatial global features, outputting a feature X with hidden temporal information. softmax Feature X softmax The outputs are sent to the decoder and the mask decoder, respectively. The mask decoder is a semantically enhanced mask decoder, which modifies the input feature X. softmax First, based on the feature pyramid network, an iterative top-down and bottom-up multi-scale feature fusion strategy is adopted to integrate feature information at different levels. Then, a semantic enhancer is combined to enrich the contextual information of the mask features using global features. At the same time, with the help of the spatial attention mechanism, the key region features related to anomaly detection in the image are selectively enhanced. The decoder is a dual Transformer instance decoder, which processes the input feature X. softmax The local instance decoder refines the query through spatial local features and supplements spatial details; the global instance decoder updates the object query using spatial global features, captures high-level semantics and coarse localization, so that the spatial global features not only have semantic information of the current frame, but also refer to the hidden temporal information in historical frames; the two are combined to generate an accurate instance-level segmentation mask to achieve object-level anomaly recognition. A query passing mechanism and a kernel reuse mechanism are also introduced during the temporal querying and decoding of consecutive frames. The query passing mechanism involves the global instance decoder reusing the target query of consecutive frames to perform cross-spatial and temporal correlation of the input global features and verify the temporal coherence. t The target query is directly used as a frame Initial frame lookup reduces redundant computation and improves timing consistency; The kernel reuse mechanism is that the global instance decoder utilizes the temporal consistency of the segmentation kernels generated by object queries across multiple frames. The segmentation kernels used in the key frame calculation are reused in subsequent frames to obtain hidden temporal information.

2. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, Furthermore, a self-attention layer is introduced based on the feature extractor to model the feature dependency relationship between the spatial and channel dimensions, thereby enhancing the extraction of effective features; The spatial local features extracted by the feature extractor include multi-scale local features X3, X4, and X5, where X3 is a low-level feature containing detailed texture, and X4 and X5 are mid-to-high-level features containing semantic information. The extracted spatial global features are global features X6 containing high-level semantic information, which are used to guide instance decoding; LSTM or GRU modules perform temporal feature enhancement on X3 to X6 respectively, hiding the temporal information into feature X. softmax In the middle, the output features X with hidden temporal information are... softmax The timing enhancement features, after being processed by the LSTM or GRU module, are used for subsequent mask decoding.

3. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The decoding of the global instance decoder includes: Feature X softmax Enter Object Queries Q and Global Featurers X respectively. G In this context, Object QueriesQ serves as a learnable instance query vector, decoding target instance information from features; Global Featurers X... G Decode the global semantic features so that the global features not only have the semantic information of the current frame, but also refer to the hidden information in the historical frames for anomaly identification. Its output is used as the key and value of the decoder to update the global context of the query vector. Using the Global Instance Decoder as the global Transformer decoder, it receives the Q and X outputs from both. G The output includes an instance query and an instance-level segmentation mask containing global semantics. The decoding of the local instance decoder includes: Feature X softmax Enter Local Features X L Get from X 3 ~ X5's local detail features X L As the key and value of the decoder, it is used to supplement the spatial details of the query vector; the Local Instance Decoder is used as a local Transformer decoder, receiving the output of the global decoder and the local detail features X. L Optimize local details of the query vector; The dual Transformer instance decoder uses output extraction as its output module to generate instance kernels and classification scores from the optimized query vector.

4. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The semantic enhancement mask decoder performs semantic enhancement through a semantic enhancer (SE). The semantic enhancer (SE) uses the hidden temporal information output by the LSTM or GRU module to strengthen the semantic information of the mask features and improve the segmentation accuracy. The output of the semantically enhanced mask decoder is X. mask It is a mask feature map that has undergone multi-scale feature fusion and SE enhancement, and is used to generate an instance segmentation mask.

5. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The timing encoding of the LSTM module is as follows: By employing gating mechanisms involving input gates, forget gates, and output gates, contextual information across frames is selectively preserved and updated to capture long-term temporal dependencies. The capture of contextual information is achieved using the following formula: ； in , , These are the forget gate, input gate, and output gate, respectively. This is a cell state used to maintain long-term dependencies across video frames, and to store short-term temporal information for transfer between LSTM units. For MobileInst at time step Extracted features; For candidate memory cell states, activate them using the tanh function, making their values ​​(-1, 1); tanh is the hyperbolic tangent activation function, which maps input values ​​to the (0,1) interval to scale candidate cell states and cell states; σ is the sigmoid activation function, which maps input values ​​to the (0,1) interval and is often used in gating mechanisms to represent the degree of gate opening; h t yes t The hidden state of a time step is used to pass information to the next time step, and also serves as the output of the current time step; h t-1 yes t The hidden state at time -1 carries information from the previous time step; C t-1 It represents the cell state at the previous moment; b f , b i , b o , b C These are the bias vectors used in the calculation of the forget gate, input gate, output gate, and candidate cell state, respectively, which are learned during the training process. W f , W i , W o , W C These are the weight matrices used in the calculation of the forget gate, input gate, output gate, and candidate cell state, respectively, which are learned and adjusted during training using the backpropagation algorithm. Final hidden state Using the fully connected classification head of the global instance decoder, a linear transformation is applied to the hidden state. The data is mapped to the same dimension as the number of categories n, and processed using the softmax function. The results are determined based on the probability values ​​of different abnormal behaviors, and the one with the highest probability value is selected as the abnormal behavior identification result.

6. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The GRU module uses only update and reset gates to manage the flow of information in the time dimension, and implements time-series coding using the following formula: ； in To update the door, control how much of the previous hidden state is retained; To reset the door, determine how much past information to use to integrate with the new information; The current hidden state; For MobileInst at time step Extracted features; σ is the Sigmoid activation function, which maps the linear transformation result to (0,1) to achieve gating on / off control; To update the weight matrix of the gate, used for... Perform a linear transformation to match the dimensions of the input and output; To reset the weight matrix of the gate, for Perform a linear transformation; The hidden state from the previous time step is concatenated with the input of the current frame; Wh It is the weight matrix of the candidate states, used to... Perform a linear transformation; For element-wise multiplication between the two, use a reset gate to filter historical hidden states, r t The closer a value is to 1, the more information in the hidden state is retained; the closer a value is to 0, the less information is retained. bz , br、bh These are the bias vectors for the update gate, reset gate, and candidate state, respectively, used to adjust the linear transformation results; It is a new state generated by combining historical information filtered by the reset gate with the current input information, providing new information for updating the hidden state, including dimensions and... Same; tanh is the hyperbolic tangent activation function, which maps the input to the range (-1, 1); After temporal encoding by the GRU module, the output contains hidden states containing spatiotemporal fusion features of the entire sequence. The data is input into a classification head located in the global instance decoder, which consists of multiple fully connected layers and activation functions. The fully connected layers achieve feature dimensionality reduction and nonlinear transformation through learnable weight matrices. The activation functions probabilize the output and map the output probability distribution. Finally, the abnormal behavior in the video sequence is determined based on the probability values.

7. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The query passing mechanism is implemented when the global instance decoder performs a temporal query on the input global features to capture the long-term interaction relationships between different targets in the global scene, and it passes through frames. Generate target query Then, local features are received and interactively generated through cross-attention. t Frame instance segmentation mask; In the When querying a frame, connect the frames. t With frames Query reuse between frames allows the target query of frame t to be passed to frame t. ， Thus, the target query As a frame The initial query, then receiving frames Local feature generation Frame instance segmentation mask.

8. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The kernel reuse mechanism is that the global instance decoder generates a dynamic convolutional kernel for instance segmentation based on the global features of the input frame. Let the frame... generate Dynamic convolution kernels Each core corresponds to the feature pattern of one instance; In processing the first frame At that time, the frame Generated dynamic convolution kernel K t Passed to frame Using convolution kernels Convolution operations are performed on local feature maps to extract hidden states; The feature map of the reuse kernel is then input into the local instance head, and the original score of each pixel belonging to the target is output. Then, the original score is compressed into a probability between 0 and 1 by the sigmoid function to obtain the final segmentation mask.

9. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 1, characterized in that, The recognition model also employs the following loss function: ； in, , To predict probabilities, Weighting coefficients for specific behavior types. For focusing parameters; , For predicting probabilities Compared with the true value The difference between them; , It represents the total number of possible abnormal behaviors. These are real labels, represented by 0 or 1; These are hyperparameters adjusted based on empirical verification of performance.

10. The lightweight temporal anomaly behavior recognition method for edge computing as described in claim 9, characterized in that, The training of the constructed recognition model includes the following operations: Multi-scale features are extracted from the MobileInst backbone network, and video data frames are processed through the temporal module LSTM or GRU module to generate inter-frame hidden information. Global features and object queries are processed by dual Transformer decoders to generate kernels. The kernels are then passed through the query and reused in the local decoder. The local features are combined to generate preliminary results. Finally, the semantically enhanced mask decoder refines the output to produce a semantically meaningful mask, thus completing the forward inference. Then, the classification loss function Focal Loss L is used. focal Constraint instance classification, Binary Cross-Entropy Loss function L BCE Mask loss is used to optimize pixel matching, and the Smooth L1 Loss function L is used. Smooth L1 Regression loss improves coordinate accuracy, and then the total loss is fused according to weights; Finally, the error is backpropagated and the parameters of the backbone network, Transformer layer, and temporal module are updated sequentially. The process is iterated until the loss converges, thus achieving a closed loop in the training of the recognition model.

Images