An abnormal behavior recognition method, device and system based on multi-modal interaction

Abstract

This invention provides a method, device, and system for abnormal behavior recognition based on multimodal interaction. It simultaneously extracts RGB sequences and human skeleton sequences from surveillance videos, performing feature extraction and alignment respectively. Second-order interaction features between the two modalities are calculated using compact bilinear pooling, and a lightweight Transformer encoder is used for global spatiotemporal correlation modeling to achieve two-stage deep fusion. Simultaneously, a feature pyramid module with a three-level structure (local, regional, and global) is designed to extract multi-scale spatiotemporal features, which are then dynamically fused using an adaptive attention mechanism. Finally, multi-level features are aggregated for classification decisions. This invention integrates the complementary advantages of appearance and structural information, achieving deep cross-modal interaction and multi-scale understanding, significantly improving the accuracy and robustness of behavior recognition in complex scenarios, and is particularly suitable for public security monitoring scenarios such as rail transit.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of behavior recognition in complex monitoring scenarios, and more specifically, to an abnormal behavior recognition method, device and system based on multimodal interaction. Background Technology

[0002] In megacities with high population density, the enclosed and complex spaces formed by the daily passenger flow of tens of millions of people in subways and stations pose a severe challenge to the accuracy and robustness of security monitoring technologies.

[0003] In subway scenarios, abnormal behavior recognition is crucial for public safety. It needs to meet the requirements of non-contact, long-distance detection to achieve monitoring and real-time early warning in dense crowds. A patent application (CN112906604A) discloses a behavior recognition method, system, and system based on skeletal and RGB frame fusion. Specifically, it discloses using an RGB camera to acquire RGB video of human movements and a depth camera to acquire depth images and 3D skeletal data to establish a behavior recognition dataset; preprocessing the behavior recognition dataset; extracting skeletal data features using a spatiotemporal graph convolutional neural network (ST-GCN) and extracting RGB frame data features using a lightweight GhostNet; constructing a network for interaction between the skeleton and RGB frames at the feature level; training the network with a training set and optimizing parameters to obtain a behavior recognition network based on skeletal and RGB frame fusion; inputting the human movement information to be recognized into the behavior recognition network for prediction and outputting the corresponding action category.

[0004] Although the aforementioned patent discloses a method for recognition based on skeleton and RGB, which utilizes a bidirectional Hadamard product correction mechanism to achieve modal complementarity, its purpose is to improve recognition performance without excessively increasing the computational burden, making it suitable for a wider range of real-time behavior recognition applications. However, due to its simple scheme and emphasis on lightweight design, its interaction depth is limited, and it does not consider the spatiotemporal dimension during fusion. It is suitable for general action recognition scenarios with high real-time requirements, but lacks dynamic adjustment. When applied to scenarios such as subway stations, the error is large, making it difficult to meet the needs of high-precision recognition. Summary of the Invention

[0005] This invention provides a method, apparatus, and system for identifying abnormal behavior based on multimodal interaction, which at least solves the problem of how to accurately identify behavior in complex scenarios in related technologies.

[0006] According to an embodiment of the present invention, an abnormal behavior recognition method based on multimodal interaction is provided, comprising the following steps:

[0007] S1. Obtain the target video sequence to be identified, and extract the RGB image sequence and the corresponding human skeleton key point sequence from it;

[0008] S2. Extract appearance features from the RGB image sequence to obtain an RGB feature sequence; extract structural features from the human skeleton key point sequence to obtain a skeleton feature sequence;

[0009] S3. Perform two-stage cross-modal feature fusion on the RGB feature sequence and the skeletal feature sequence to obtain deep fused features;

[0010] The dual-stage cross-modal feature fusion includes:

[0011] First-stage fusion: Based on the compact bilinear pooling method, the second-order interaction features between the RGB feature sequence and the skeletal feature sequence are calculated to obtain the initial fusion features;

[0012] The second stage of fusion involves combining the initial fusion features, the RGB feature sequence, and the skeletal feature sequence to form a multimodal feature sequence, which is then input into the Transformer encoder for correlation modeling, and the deep fusion features are output.

[0013] S4. Perform multi-scale spatiotemporal feature extraction on the RGB feature sequence and the skeletal feature sequence respectively to obtain RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features;

[0014] The multi-scale spatiotemporal feature extraction includes at least local scale feature extraction, regional scale feature extraction, and global scale feature extraction.

[0015] S5. Perform adaptive fusion of the RGB multi-scale spatiotemporal features and the skeletal multi-scale spatiotemporal features based on an attention mechanism to obtain adaptive spatiotemporal fusion features;

[0016] S6. Based at least on the deep fusion features and the adaptive spatiotemporal fusion features, perform feature integration and classification, and output the abnormal behavior recognition result corresponding to the target video sequence.

[0017] Further, in step S2, the extraction of appearance features from the RGB image sequence specifically includes:

[0018] The ResNet-18 convolutional neural network is used to extract features from each frame of the RGB image, and the spatial dimension is compressed by spatial pooling to form the RGB feature sequence.

[0019] Furthermore, in step S2, the extraction of structural features from the human skeletal key point sequence specifically includes:

[0020] The GraphSAGE convolutional neural network is used to process the human skeletal key point sequence to extract the spatiotemporal correlation features between joints, thereby obtaining the skeletal feature sequence; and further, the channel dimension of the skeletal feature sequence is aligned with the RGB feature sequence through linear projection.

[0021] Furthermore, specifically, the self-attention module in the Transformer encoder independently applies a channel attention mechanism to each time frame to reduce computational complexity.

[0022] Furthermore, the multi-scale spatiotemporal feature extraction described in step S4 specifically includes:

[0023] For the RGB feature sequence, it is divided into multiple blocks along the spatial dimension; for the skeletal feature sequence, it is divided into units of joints.

[0024] For the segmented features, perform the following operations respectively:

[0025] Local scale feature extraction: Feature transformation is performed using a combination of 1×1 spatial convolution and 3×1 temporal convolution;

[0026] Regional scale feature extraction: 3×3 spatial convolution is used for feature transformation to expand the spatial receptive field;

[0027] Global-scale feature extraction: First, global features are generated through pooling operations. Then, the weights of different regions are calculated using a self-attention mechanism, and the global features are weighted and fused.

[0028] Furthermore, step S5, which describes the adaptive fusion of RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features based on an attention mechanism, specifically includes:

[0029] S51. Map the dimensions of the skeletal multi-scale spatiotemporal features through a fully connected layer to align the number of spatial blocks with the RGB multi-scale spatiotemporal features;

[0030] S52. Using the aligned RGB multi-scale spatiotemporal features as the query vector and the aligned skeletal multi-scale spatiotemporal features as the key vector and value vector, input them into the multi-head attention module to calculate the weighted skeletal features.

[0031] S53. The weighted skeletal features are concatenated with the RGB multi-scale spatiotemporal features, and then fused through a fully connected layer to generate the adaptive spatiotemporal fusion features.

[0032] Furthermore, step S6 specifically includes:

[0033] The deep fusion feature, the adaptive spatiotemporal fusion feature, the aggregated feature of the original RGB feature sequence obtained from step S2, and the aggregated feature of the original skeletal feature sequence are mapped through a fully connected layer and then concatenated to form a global fusion feature.

[0034] After pooling and dimensionality reduction of the global fusion features, the data is input into a classifier to obtain the abnormal behavior recognition results.

[0035] According to another embodiment of the present invention, an abnormal behavior recognition system based on multimodal interaction is provided, comprising:

[0036] The data acquisition and preprocessing module is used to acquire the target video sequence to be identified and extract the RGB image sequence and the corresponding human skeleton key point sequence from it.

[0037] A dual-modal feature extraction module is used to extract appearance features from the RGB image sequence to obtain an RGB feature sequence; and to extract structural features from the human skeleton key point sequence to obtain a skeleton feature sequence.

[0038] A two-stage cross-modal fusion module is used to perform two-stage cross-modal feature fusion on the RGB feature sequence and the skeletal feature sequence to obtain deep fused features; the module includes a compact bilinear pooling unit for calculating second-order interaction features, and a lightweight Transformer encoder for performing cross-modal spatiotemporal correlation modeling.

[0039] The multi-scale spatiotemporal feature extraction module is used to extract spatiotemporal features at local, regional, and global scales from the RGB feature sequence and the skeletal feature sequence, respectively, to obtain RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features.

[0040] An adaptive spatiotemporal fusion module is used to adaptively fuse the RGB multi-scale spatiotemporal features and the skeletal multi-scale spatiotemporal features based on an attention mechanism to obtain adaptive spatiotemporal fusion features;

[0041] The feature integration and classification module is used to integrate and classify features based on at least the deep fusion features and the adaptive spatiotemporal fusion features, and output the abnormal behavior recognition result corresponding to the target video sequence.

[0042] According to another embodiment of the present invention, a computer-readable storage medium for an abnormal behavior recognition method based on multimodal interaction is also provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments at runtime.

[0043] According to another embodiment of the present invention, an electronic system for an abnormal behavior recognition method based on multimodal interaction is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0044] This invention effectively overcomes the inherent limitations of single-modality systems by deeply fusing appearance and structural information, enabling the system to perform stably in complex real-world scenarios such as varying lighting and partial occlusion. Simultaneously, the dual-stage fusion strategy allows the model to not only utilize bimodal information but also gain a deeper understanding of how these information dynamically relate to each other. The hierarchical multi-scale extraction mechanism allows the model to simultaneously analyze microscopic, mesoscopic, and macroscopic behavioral patterns. An adaptive attention mechanism ensures that the fusion strategy is optimal for different input samples. Addressing practical deployment needs, key modules have been designed with lightweight features. Through frame-level channel attention compression and other strategies, computational complexity and the number of parameters are effectively controlled while maintaining performance, meeting the real-time analysis requirements of security monitoring systems. Attached Figure Description

[0045] Figure 1 This is a flowchart of an abnormal behavior recognition method based on multimodal interaction according to an embodiment of the present invention;

[0046] Figure 2 This is a structural block diagram of an abnormal behavior recognition system based on multimodal interaction according to an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0048] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0049] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0050] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0051] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0052] It should be noted that the "multimodal interaction" mentioned in this application specifically refers to the bidirectional information flow and semantic alignment between RGB features and skeletal features through attention mechanisms (such as Transformer and adaptive spatiotemporal attention modules); while "feature fusion" generally refers to combining features from different sources through methods such as concatenation and weighted summation. This scheme achieves deep interaction through a "two-stage fusion strategy": the first stage (bilinear pooling) performs second-order interaction of local features, and the second stage (Transformer encoding) performs contextual interaction of global temporal sequence.

[0053] This embodiment provides an abnormal behavior recognition method based on multimodal interaction. Figure 1 This is a flowchart of an abnormal behavior recognition method based on multimodal interaction according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0054] S1. Dual-modal data synchronous acquisition and preprocessing: Input a surveillance video segment to be analyzed, and process it in parallel to obtain two aligned data streams: an RGB image sequence F_RGB∈R^(N×C1×T×H×W), where N is the batch size, C1 is the number of channels, T is the number of time frames, and H×W is the size of a single frame RGB image; and a human skeletal keypoint sequence F_SKE∈R^(N×C2×T×V) extracted from the same batch of images through a pose estimation network, where T is the number of time frames, C2 is the number of channels (i.e., joint coordinates (x,y) and confidence level), and V is the total number of joints. The skeletal data is normalized based on the coordinates of the center of the human hip joint in each frame to eliminate the influence of the absolute position of the person in the image.

[0055] S2. Preliminary extraction and alignment of heterogeneous features:

[0056] RGB Appearance Flow: A lightweight convolutional neural network (e.g., the ResNet-18 variant MC18) pre-trained on a large action recognition dataset (such as Kinetics) serves as the backbone. This network independently extracts spatial features from each frame of the RGB image and finally compresses the 2D feature map of each frame into a one-dimensional feature vector through a global spatial pooling layer (such as GAP), thus representing the entire video segment as a temporal feature sequence F_RGB'∈R^(N×C1×T×L), where C1 is the number of feature channels (128) and L is the number of spatial patches. In this embodiment, the first convolutional layer uses a 3×3 kernel with 32 channels, and the number of channels in subsequent layers increases to 128 sequentially. A Leaky ReLU activation function is applied after each convolution.

[0057] Skeletal Structure Flow: A Graph Convolutional Network (GCN) is used to process the skeletal sequence. V joints of the human body are defined as graph nodes, and skeletal connections are defined as edges. Multi-layer graph convolutional operations, such as two layers of GraphSAGE, are used to aggregate neighborhood joint information in the spatiotemporal dimensions, learn the dynamic structural features between joints, and output a skeletal feature sequence F_SKE'∈R^(N×C2×T×V). To ensure subsequent fusion, a learnable linear projection layer, Linear(C,C), is used to align the number of channels of the skeletal features to be the same as the RGB features.

[0058] S3. Two-stage deep cross-modal fusion: This stage is one of the core innovations of this invention, aiming to achieve progressive fusion from local fine-grained interaction to global context modeling.

[0059] Phase 1: Fine-grained second-order interaction based on compact bilinear pooling. To efficiently capture the pairwise correlation between RGB features F_RGB' and skeletal features F_SKE', this step uses compact bilinear pooling to calculate their second-order interaction feature F_bilinear. The specific implementation includes:

[0060] 1. Feature initialization: Perform nonlinear transformations on the input features to enhance their expressive power.

[0061] F_RGB_tilde=Sigmoid(F_RGB'), F_SKE_tilde=ReLU(F_SKE')

[0062] 2. Random Projection and Fourier Transform: Generate two fixed random projection matrices W_a and W_b, whose elements follow a standard normal distribution, with dimensions [d, C]. For eigenvectors f_r and f_s, calculate the Fourier transform of their projected sign functions:

[0063] Psi(f_r)=FFT(sign(W_a*f_r)), Psi(f_s)=FFT(sign(W_b*f_s))

[0064] Where sign(·) is the sign function and FFT(·) represents the Fast Fourier Transform.

[0065] 3. Interactive Feature Calculation: The outer product features are approximated by the inverse Fourier transform.

[0066] f_bp=IFFT(Psi(f_r)⊙Psi(f_s))

[0067] Here, ⊙ represents element-wise multiplication, and IFFT(·) represents the inverse fast Fourier transform.

[0068] 4. Output: Perform the above calculations at all positions in the spatiotemporal dimension to obtain a compact bilinear feature F_bilinear with dimensions [d, T]. Preferably, d = 512.

[0069] The second stage: Global spatiotemporal correlation modeling based on a lightweight Transformer. The fine-grained interaction feature F_bilinear obtained in the first stage is concatenated with the original aligned RGB feature F_RGB' and skeletal feature F_SKE' along the feature dimension to form a multimodal joint sequence X_multi with dimensions [N,T,(C+C+d)]. To preserve sequence order information, learnable positional encoding is added. Subsequently, X_multi is fed into a lightweight Transformer encoder. The innovative lightweight design of this encoder is reflected in:

[0070] Frame-level channel attention compression module: Before entering the standard multi-head self-attention layer, the input sequence X_multi is sliced ​​by time frame. For the feature X_t of frame t (with dimension D, where D = 2C + d), the following operations are performed to compress the channel dimension and focus on important information:

[0071] a) Calculate channel statistics: z_t=GlobalAvgPool(X_t), with dimension D.

[0072] b) Generate channel weights through a two-layer bottleneck MLP: a_t=σ(W_2·δ(W_1·z_t+b_1)+b_2), where W_1 has dimensions [D / r,D], W_2 has dimensions [D,D / r], δ is the ReLU activation function, σ is the Sigmoid function, r is the compression ratio (preferably r=16), and b_1 and b_2 are bias vectors.

[0073] c) Apply weights: X'_t = a_t ⊙ X_t.

[0074] The features X'_t after processing all frames are reassembled into a new sequence X'. This operation effectively reduces the feature dimension involved in subsequent attention calculations from D to approximately D / r, thereby significantly reducing the computational complexity of self-attention from O(T^2·D) to O(T^2·D / r).

[0075] The lightweight sequence X' is then modeled for global spatiotemporal context using a standard multi-head self-attention mechanism and a feedforward network (FFN). The encoder finally outputs a deep fusion feature F_trans with dimensions [N,T,D].

[0076] S4. Hierarchical Multi-Scale Spatiotemporal Feature Pyramid Extraction: This stage is another core innovation of the present invention, which aims to explicitly capture the spatiotemporal patterns of behavior at different spatial scales.

[0077] Feature segmentation: For an RGB feature sequence F_RGB', it is divided into B consecutive blocks in the spatial (channel) dimension (e.g., a C=128 dimensional feature can be regarded as B=128 blocks). For a skeletal feature sequence F_SKE', it naturally uses V joints as the block unit.

[0078] Parallel multi-scale convolution extraction: Three parallel spatiotemporal convolution branches are applied to each block (or joint), focusing on different scales respectively:

[0079] 1. Local scale branch: A concatenated structure of 1×1 spatial convolution (used to fuse information from each channel within a block) and 3×1 temporal convolution (used to capture short-term dynamics) is used to extract fine motor features such as "finger micro-movement" and "head rotation".

[0080] 2. Region-Scale Branch: To expand the spatial receptive field, this branch simulates a 3×3 spatial convolution. In practice, the one-dimensional block feature sequence F (dimension [N,C,T,B]) is first reshaped into a tensor with a spatial grid structure using a deterministic mapping rule. For example, when B=128, it can be reshaped into F_2d with dimensions [N,C,T,8,16] (because 8×16=128). This mapping rule is preset; for example, feature indices [0:15] are mapped to the first row of the grid, [16:31] to the second row, and so on. After reshaping, a 3×3 2D convolution is performed on the imaginary (8,16) grid to capture collaborative motion features involving multiple adjacent body parts, such as "arm waving" or "leg strides." The processed sequence is then restored to a one-dimensional sequence form.

[0081] 3. Global Scale Branch: This branch aims to obtain overall behavioral patterns. First, global average pooling (GAP) and global max pooling (GMP) are performed on the features respectively, and the results are summed to obtain a global descriptor. Then, a lightweight self-attention (SA) module is used to calculate the contribution weights of different blocks to the global descriptor, and weighted aggregation is performed to highlight key regions and capture systemic behavioral features such as "falling down due to loss of balance" and "running fast".

[0082] The output features of the three branches are concatenated along the channel dimension and max pooling is applied along the time dimension to obtain the RGB multi-scale feature S_C (dimension [N,C_m,B]) and the skeletal multi-scale feature S_K (dimension [N,C_m,V]), where C_m is the total number of channels of the multi-scale feature.

[0083] S5. Adaptive Spatiotemporal Feature Fusion: This module aims to dynamically and selectively fuse multi-scale spatiotemporal features from RGB and skeleton.

[0084] Dimension alignment: Since B and V are usually not equal, the spatial dimensions of the skeletal multi-scale feature S_K are first mapped from V to B through a fully connected layer Linear(3V,3B) to obtain the aligned feature S_K'.

[0085] Cross-attention fusion: The RGB multi-scale feature S_C is used as the query, and the aligned skeletal feature S_K' is used as the key and value. Semantic association weights are calculated using a multi-head attention mechanism. For the i-th head, the calculation process is as follows:

[0086] Head_i=Attention(S_C*W_i^Q,S_K'*W_i^K,S_K'*W_i^V)=Softmax((S_C*W_i^Q)*(S_K'*W_i^K)^T / sqrt(d_k))*(S_K'*W_i^V)

[0087] Where W_i^Q, W_i^K, and W_i^V are learnable projection matrices with dimensions [C_m, d_k], where d_k is a scaling factor (usually d_k = C_m / h), and h is the number of attention heads. The outputs of all heads are concatenated and linearly projected to obtain the initial fused output Attn_out.

[0088] Attn_out=Concat(Head_1,...,Head_h)*W^O

[0089] Where W^O is the output projection matrix with dimensions [h*d_k,C_m].

[0090] Global bias enhancement: To further stabilize the fusion process and introduce global structural information, the global average value of the skeletal feature S_K' is calculated as a bias term: bias=Mean(S_K',dim=[spatial]), with dimensions [N,C_m]. This bias term is added to the output of the cross-attention module.

[0091] Finally, the enhanced attention output is concatenated with the original RGB multi-scale feature S_C, and then passed through a fully connected fusion layer to generate the final adaptive spatiotemporal fusion feature ST_fst with dimensions [N, D_f].

[0092] S6. Global Feature Aggregation and Classification Decision: Integrates rich information from different levels and modules of the network to make the final decision.

[0093] Global features are extracted from the original RGB stream and the skeleton stream respectively (e.g., average pooling is performed on F_RGB' and F_SKE' along the time and spatial dimensions).

[0094] The two global features, the deep fusion feature F_trans (after time pooling), and the adaptive spatiotemporal fusion feature ST_fst are respectively dimensionality-reduced and refined through independent fully connected layers with non-linear activation.

[0095] All refined feature vectors are concatenated to form a comprehensive global description vector F_global.

[0096] To reduce computational cost and prevent overfitting, F_global is divided into several segments, and each segment is averaged before being reassembled.

[0097] The processed features are fed into a fully connected classifier, which outputs the probability distribution corresponding to each type of abnormal behavior.

[0098] Specifically, one possible implementation method is as follows:

[0099] This embodiment aims to identify six typical abnormal behaviors of subway platform passengers (falling down, fighting, climbing over turnstiles, going in the wrong direction, abnormal lingering, and throwing objects from heights).

[0100] Step S101: Data preparation and model input.

[0101] A dedicated dataset was constructed by collecting actual surveillance videos from subway stations in multiple cities. The videos were segmented into 2-second segments (corresponding to T=50 frames, original frame rate 25fps). During training, each frame was uniformly downsampled to H×W=256×256. The high-precision open-source pose estimator AlphaPose was used to process each frame, extracting V=17 standard 2D human keypoints (x, y coordinates and confidence scores) to form a skeletal sequence. For the skeletal data of each segment, coordinate normalization was performed with the hip joint center of the first frame as the origin.

[0102] Step S102: Specific configuration of the dual-stream feature extraction network.

[0103] RGB Branch: A ResNet-18 pre-trained on the Kinetics-400 dataset is used as the backbone. Its original classification head is removed, retaining only the output up to layer 4. Input is F_RGB, and the network outputs an 8×8 feature map with C=256 channels. Global average pooling is then applied to compress each frame's feature map into a 256-dimensional vector, resulting in F_RGB'.

[0104] Skeletal Branches: A two-layer GraphSAGE network is constructed, with each layer followed by batch normalization and LeakyReLU activation. The graph structure uses the natural human skeleton for connections. The first layer maps the input dimension 3 to 64, and the second layer maps it to 256, aligning with the RGB branches. Therefore, the output is 'F_SKE'.

[0105] Step S103: Specific implementation of two-stage deep cross-modal fusion.

[0106] Compact bilinear pooling: Set the output dimension d = 512. Generate fixed random matrices W_a and W_b according to the aforementioned mathematical process. For each time step t, take f_r = F_RGB'[:,t] and f_s = mean(F_SKE'[:,t,:],dim = -1). Calculate f_bp using the FFT method described in S3. Repeat for all t to obtain F_bilinear.

[0107] Sequence construction and positional encoding: Concatenate F_RGB', F_SKE', and F_bilinear along the channel dimension to obtain X_multi. Add a learnable positional encoding matrix PE to obtain X = X_multi + PE.

[0108] Lightweight Transformer encoder:

[0109] Set the Transformer encoder layer number L=4, the multi-head attention head number h=8, and the feedforward network hidden layer dimension to 4096.

[0110] Frame-level channel attention compression: For input X, channel attention is applied independently to each column as described in S3. Compression ratio r = 16.

[0111] X' is input into the subsequent L-layer standard Transformer encoder. Finally, the feature at the CLS token position of the last layer output is taken, or the features from all time steps are averaged and pooled to obtain the deep fusion feature F_trans.

[0112] Step S104: Extraction of multi-scale spatiotemporal feature pyramid.

[0113] Block segmentation: The 256 channels of the RGB feature F_RGB' are considered as B=256 blocks. The skeletal feature F_SKE' is divided into blocks of 17 joints.

[0114] Three-level parallel convolution (taking the RGB branch as an example):

[0115] Local module: Conv1d(256,128,kernel_size=1)->LeakyReLU->Conv1d(128,128,kernel_size=3,padding=1,groups=128)

[0116] Region module: Reshapes the features to [256, 50, 16, 16]. Uses Conv2d(256, 128, kernel_size=3, padding=1, groups=256) operation. The output is then reshaped back to [128, 50, 256].

[0117] Global modules: Path A: GAP over spatial -> Linear(256,128); Path B: GMP overspatial -> Linear(256,128); after addition, they are passed through a MultiHeadAttention(embed_dim=128,num_heads=4) module.

[0118] The outputs of the three modules are concatenated along the channel dimension (128+128+128=384) to obtain S_C_temp. Max pooling is then performed along the time dimension to obtain S_C.

[0119] Step S105: Adaptive spatiotemporal feature fusion.

[0120] Dimension alignment: S_K is obtained by passing through the Linear(17,256) layer to obtain S_K'.

[0121] Cross-attention: With S_C as Q, S_K' as K and V. Set h=4, d_k=384 / 4=96. Calculate the multi-head attention output Attn_out according to the aforementioned formula.

[0122] Bias calculation and fusion: Calculate bias = mean(S_K', dim = 1). Attn_out_enhanced = Attn_out + bias.unsqueeze(-1).

[0123] Final fusion: ST_fst=Linear(Concat(S_C,Attn_out_enhanced)), output dimension D_f=512.

[0124] Step S106: Global decision-making, classification, and training.

[0125] Extract the global features of the original branch: f_rgb_g=GAP(F_RGB'), f_ske_g=GAP(GAP(F_SKE')).

[0126] f_rgb_g, f_ske_g, F_trans, and ST_fst are passed through four independent Linear(*, 256) layers.

[0127] By concatenating four 256-dimensional vectors, we obtain F_global.

[0128] F_global is divided into P=16 blocks, each with 64 dimensions. Average pooling is performed on each block to obtain 16 dimensions, and then the blocks are concatenated back to 256 dimensions.

[0129] Finally, the probability of the six types of abnormal behavior is output through the Linear(256,6) classification layer.

[0130] Training and Loss Function: The goal of model training is to minimize the following multi-task joint loss function L:

[0131] L=L_CE+λ*L_reg

[0132] Where L_CE is the standard multi-class cross-entropy loss: L_CE=-(1 / N)*Σ_i Σ_c y_{i,c} * log(p_{i,c})

[0133] L_reg is the L2 weight regularization term: L_reg = Σ_θ ||θ||_2^2

[0134] λ is the regularization coefficient, which is set to 1e-4 in this embodiment.

[0135] Training details: The AdamW optimizer was used with an initial learning rate of 3e-4 and weight decay of 1e-4. Cosine annealing was employed for learning rate scheduling. The batch size was 32, and the training lasted for 150 epochs. Data augmentation was performed using random horizontal flipping, temporal pruning, and color dithering.

[0136] Those skilled in the art will understand that the above embodiments are merely for illustrating the present invention more clearly and are not intended to be limiting. Various substitutions and modifications can be made without departing from the principles of the present invention, for example:

[0137] The RGB backbone network can be replaced with more efficient architectures such as EfficientNet and ConvNeXt. The skeletal processing network can be replaced with more advanced spatiotemporal graph convolutional networks such as CTR-GCN and MS-G3D. In compact bilinear pooling, other functions such as multinomial kernels can be used to approximate higher-order interactions. The spatial convolution kernel size and dilation rate of the "region modules" in multi-scale extraction can be adjusted according to the specific scene. The attention mechanism in adaptive fusion can be replaced with gated attention, dynamic convolution, etc.

[0138] These modifications and substitutions all fall within the scope of protection of the present invention as defined by the appended claims.

[0139] This embodiment effectively overcomes the inherent limitations of single-modality systems by deeply fusing appearance and structural information, enabling the system to perform stably in complex real-world scenarios such as varying lighting and partial occlusion. Simultaneously, the dual-stage fusion strategy allows the model to not only utilize bimodal information but also gain a deeper understanding of how these information dynamically relate to each other. The hierarchical multi-scale extraction mechanism allows the model to simultaneously analyze microscopic, mesoscopic, and macroscopic behavioral patterns. An adaptive attention mechanism ensures that the fusion strategy is optimal for different input samples. To meet practical deployment requirements, key modules have been designed with lightweight features. Through strategies such as frame-level channel attention compression, computational complexity and the number of parameters are effectively controlled while maintaining performance, meeting the real-time analysis requirements of security monitoring systems.

[0140] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0142] This embodiment also provides an abnormal behavior recognition system based on multimodal interaction. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0143] Figure 2 This is a structural block diagram of an abnormal behavior recognition system based on multimodal interaction according to an embodiment of the present invention, such as... Figure 2 As shown, the system includes:

[0144] The data acquisition and preprocessing module is used to acquire the target video sequence to be identified and extract the RGB image sequence and the corresponding human skeleton key point sequence from it.

[0145] The dual-modal feature extraction module is used to extract appearance features from RGB image sequences to obtain RGB feature sequences; and to extract structural features from human skeletal key point sequences to obtain skeletal feature sequences.

[0146] The two-stage cross-modal fusion module is used to perform two-stage cross-modal feature fusion on RGB feature sequences and skeletal feature sequences to obtain deep fused features. The module includes a compact bilinear pooling unit for computing second-order interaction features and a lightweight Transformer encoder for cross-modal spatiotemporal correlation modeling.

[0147] The multi-scale spatiotemporal feature extraction module is used to extract spatiotemporal features at the local, regional, and global scales from the RGB feature sequence and the skeletal feature sequence, respectively, to obtain RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features.

[0148] The adaptive spatiotemporal fusion module is used to adaptively fuse RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features based on an attention mechanism to obtain adaptive spatiotemporal fusion features.

[0149] The feature integration and classification module is used to integrate and classify features based on at least deep fusion features and adaptive spatiotemporal fusion features, and output the abnormal behavior recognition results corresponding to the target video sequence.

[0150] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0151] Embodiments of the present invention also provide a computer-readable storage medium for an abnormal behavior recognition method based on multimodal interaction, wherein the computer-readable storage medium stores a computer program configured to execute the steps in any of the above method embodiments at runtime.

[0152] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0153] Embodiments of the present invention also provide an electronic system for an abnormal behavior recognition method based on multimodal interaction, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0154] In one exemplary embodiment, the electronic system may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0155] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0156] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division of an abnormal behavior recognition method based on multimodal interaction. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of systems or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying abnormal behavior based on multimodal interaction, characterized in that, Includes the following steps: S1. Obtain the target video sequence to be identified, and extract the RGB image sequence and the corresponding human skeleton key point sequence from it; S2. Extract appearance features from the RGB image sequence to obtain an RGB feature sequence; Structural features are extracted from the key point sequence of the human skeleton to obtain a skeleton feature sequence; S3. Perform two-stage cross-modal feature fusion on the RGB feature sequence and the skeletal feature sequence to obtain deep fused features; The dual-stage cross-modal feature fusion includes: First-stage fusion: Based on the compact bilinear pooling method, the second-order interaction features between the RGB feature sequence and the skeletal feature sequence are calculated to obtain the initial fusion features; The second stage of fusion involves combining the initial fusion features, the RGB feature sequence, and the skeletal feature sequence to form a multimodal feature sequence, which is then input into the Transformer encoder for correlation modeling, and the deep fusion features are output. S4. Perform multi-scale spatiotemporal feature extraction on the RGB feature sequence and the skeletal feature sequence respectively to obtain RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features; The multi-scale spatiotemporal feature extraction includes at least local scale feature extraction, regional scale feature extraction, and global scale feature extraction. S5. Perform adaptive fusion of the RGB multi-scale spatiotemporal features and the skeletal multi-scale spatiotemporal features based on an attention mechanism to obtain adaptive spatiotemporal fusion features; S6. Based at least on the deep fusion features and the adaptive spatiotemporal fusion features, perform feature integration and classification, and output the abnormal behavior recognition result corresponding to the target video sequence.

2. The method according to claim 1, characterized in that, In step S2, the extraction of appearance features from the RGB image sequence specifically includes: The ResNet-18 convolutional neural network is used to extract features from each frame of the RGB image, and the spatial dimension is compressed by spatial pooling to form the RGB feature sequence.

3. The method according to claim 1, characterized in that, In step S2, the extraction of structural features from the human skeletal key point sequence specifically includes: The GraphSAGE convolutional neural network is used to process the human skeletal key point sequence to extract the spatiotemporal correlation features between joints, thereby obtaining the skeletal feature sequence; and further, the channel dimension of the skeletal feature sequence is aligned with the RGB feature sequence through linear projection.

4. The method according to claim 1, characterized in that, Specifically, it includes: The self-attention module in the Transformer encoder applies channel attention independently to each time frame to reduce computational complexity.

5. The method according to claim 1, characterized in that, The multi-scale spatiotemporal feature extraction described in step S4 specifically includes: For the RGB feature sequence, it is divided into multiple blocks along the spatial dimension; for the skeletal feature sequence, it is divided into units of joints. For the segmented features, perform the following operations respectively: Local scale feature extraction: Feature transformation is performed using a combination of 1×1 spatial convolution and 3×1 temporal convolution; Regional scale feature extraction: 3×3 spatial convolution is used for feature transformation to expand the spatial receptive field; Global-scale feature extraction: First, global features are generated through pooling operations. Then, the weights of different regions are calculated using a self-attention mechanism, and the global features are weighted and fused.

6. The method according to claim 1 or 5, characterized in that, Step S5, which describes the adaptive fusion of RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features based on an attention mechanism, specifically includes: S51. Map the dimensions of the skeletal multi-scale spatiotemporal features through a fully connected layer to align the number of spatial blocks with the RGB multi-scale spatiotemporal features; S52. Using the aligned RGB multi-scale spatiotemporal features as the query vector and the aligned skeletal multi-scale spatiotemporal features as the key vector and value vector, input them into the multi-head attention module to calculate the weighted skeletal features. S53. The weighted skeletal features are concatenated with the RGB multi-scale spatiotemporal features, and then fused through a fully connected layer to generate the adaptive spatiotemporal fusion features.

7. The method according to claim 1, characterized in that, Step S6 specifically includes: The deep fusion feature, the adaptive spatiotemporal fusion feature, the aggregated feature of the original RGB feature sequence obtained from step S2, and the aggregated feature of the original skeletal feature sequence are mapped through a fully connected layer and then concatenated to form a global fusion feature. After pooling and dimensionality reduction of the global fusion features, the data is input into a classifier to obtain the abnormal behavior recognition results.

8. An abnormal behavior recognition system based on multimodal interaction, characterized in that, include: The data acquisition and preprocessing module is used to acquire the target video sequence to be identified and extract the RGB image sequence and the corresponding human skeleton key point sequence from it. A dual-modal feature extraction module is used to extract appearance features from the RGB image sequence to obtain an RGB feature sequence; and to extract structural features from the human skeleton key point sequence to obtain a skeleton feature sequence. A two-stage cross-modal fusion module is used to perform two-stage cross-modal feature fusion on the RGB feature sequence and the skeletal feature sequence to obtain deep fused features; the module includes a compact bilinear pooling unit for calculating second-order interaction features, and a lightweight Transformer encoder for performing cross-modal spatiotemporal correlation modeling. The multi-scale spatiotemporal feature extraction module is used to extract spatiotemporal features at local, regional, and global scales from the RGB feature sequence and the skeletal feature sequence, respectively, to obtain RGB multi-scale spatiotemporal features and skeletal multi-scale spatiotemporal features. An adaptive spatiotemporal fusion module is used to adaptively fuse the RGB multi-scale spatiotemporal features and the skeletal multi-scale spatiotemporal features based on an attention mechanism to obtain adaptive spatiotemporal fusion features; The feature integration and classification module is used to integrate and classify features based on at least the deep fusion features and the adaptive spatiotemporal fusion features, and output the abnormal behavior recognition result corresponding to the target video sequence.

9. An electronic device using an abnormal behavior recognition method based on multimodal interaction, characterized in that, include: The method for identifying multi-scale abnormal behavior based on multimodal interaction as described in any one of claims 1 to 7 includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method for identifying multi-scale abnormal behavior based on multimodal interaction as described in any one of claims 1 to 7.

10. A computer-readable storage medium for an abnormal behavior recognition method based on multimodal interaction, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multi-scale abnormal behavior recognition method based on multimodal interaction as described in any one of claims 1 to 7.

Images