A behavior recognition method based on federated learning and mamba network

Abstract

The application provides a behavior recognition method based on federated learning and Mamba network, acquires real-time video stream in a target monitoring area, extracts skeleton key point coordinate sequences of a human body in each frame image of the real-time video stream and corresponding tracking IDs; converts absolute pixel coordinates of the skeleton key point coordinate sequences into normalized relative coordinates relative to a human body center point, calculates key point displacement velocity vectors between adjacent frames, and constructs a high-dimensional space-time feature matrix; a federated learning system includes a central server and multiple local Mamba time sequence state space models; a local model is trained to obtain local model parameter updates; the central server globally aggregates all uploaded local model parameters to generate global model parameters and distribute the global model parameters to each edge client for updating; a stable model is trained to recognize and analyze the high-dimensional space-time feature matrix, and output a behavior recognition result.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a behavior recognition method based on federated learning and Mamba networks. Background Technology

[0002] As artificial intelligence technology extends to edge computing, video behavior recognition technology is playing an increasingly important role in scenarios such as home-based elderly care monitoring, school teaching assessment, and factory safety production. However, existing visual behavior recognition technology solutions generally face the following core pain points: 1. Data silos caused by privacy and security regulations: In real-world scenarios (especially homes, hospital wards, and classrooms), video streams contain a large amount of sensitive facial and behavioral privacy data. According to relevant data security regulations, raw videos are strictly prohibited from being uploaded to the cloud. This results in a single sensing node (such as a single home or classroom) only being able to acquire a very small amount of abnormal behavioral data (e.g., very few samples of "falling" or "fighting"), making the model prone to overfitting and exhibiting poor generalization ability.

[0003] 2. Computational bottleneck in long-term modeling at the edge: Human behavior (such as slow falls or prolonged wandering) is a long-term evolutionary process. The computational complexity of traditional Transformer-based time-series models increases quadratically with the sequence length, while RNN-type models suffer from long-term memory forgetting problems. Both are difficult to implement high-frame-rate real-time inference for multiple users on edge gateways or smart cameras with limited computing power and power consumption.

[0004] 3. Pose scale distortion caused by traditional object detection: Existing technologies usually rely on the center of the bounding box as a normalization reference. Once the target is partially occluded or the bounding box jitters, it will cause drastic changes in the features input to the time series model, causing model "illusion" and false alarms.

[0005] Therefore, there is an urgent need for a universal underlying technology framework that can completely isolate video privacy leaks at the physical level, break down data silos between different scenarios, and achieve high-precision temporal action understanding with extremely low computing power. Summary of the Invention

[0006] To address the problems existing in the background technology, this invention proposes a behavior recognition method based on federated learning and Mamba networks.

[0007] A behavior recognition method based on federated learning and Mamba networks includes: S100. Obtain the real-time video stream within the target monitoring area, and extract the coordinate sequence of the human skeleton key points and the corresponding tracking ID in each frame of the real-time video stream. S200. Convert the absolute pixel coordinates of the skeleton key point coordinate sequence into normalized relative coordinates relative to the human body center point, and calculate the key point displacement velocity vector between adjacent frames to construct a high-dimensional spatiotemporal feature matrix. S300: Construct a federated learning system, including a central server and multiple local Mamba time-series state-space models; S400. Use the local non-independent and identically distributed dataset to train the Mamba time-series state-space model locally, obtain the local model parameter update, and upload the updated local model parameters to the central server. The S500 central server uses an improved FedAvg algorithm with a zero-reduction smoothing term to globally aggregate all uploaded local model parameters, generate global model parameters, and distribute them to each edge client for updates. S600, repeat steps S400-S500 until the set number of communication rounds or global model convergence is reached, thereby obtaining a stable federated learning system; S700: The high-dimensional spatiotemporal feature matrix is ​​identified and analyzed using the local Mamba temporal state space model, and the behavior recognition results are output.

[0008] Based on the above, step S100 includes the following steps: S110, Input the... Frame video image Multi-scale image features are extracted using the YOLOv11 backbone network. S120. Output the human body bounding box through the detection head. and confidence level ,in For the detected target index, The coordinates of the top left corner of the bounding box. These are the width and height of the bounding box, respectively; S130. Use the BoT-SORT tracking algorithm to associate targets in adjacent frames and assign a unique tracking ID to each target. , thus obtaining the trajectory set ; S140. Regress each target using keypoint heads. of Coordinates of key points and visibility confidence ,in , , For the first Frame number The x-coordinates of the key points For the first Frame number The ordinates of the key points.

[0009] Based on the above, step S200 includes the following steps: S210, Calculation Target In the Coordinates of the human body center point in the frame image : ; in, For the goal The x-coordinate of the center point, For the goal The ordinate of the center point, For the goal The x-coordinate of the top-left corner of the bounding box. For the goal The y-coordinate of the top-left corner of the bounding box. For the goal The width of the bounding box, For the goal The height of the bounding box; S220, Set the absolute coordinates of key points Convert to normalized relative coordinates To eliminate the influence of the human body's position and scale in the image: ; in, For the first Frame number Normalized relative x-coordinates of key points For the first Frame number Normalized relative ordinates of key points The original pixel x-coordinate, The original pixel ordinate; S230. Calculate the normalized velocity vector of key points between adjacent frames. To capture the dynamic characteristics of the action: ; in, For the first Frame number The horizontal velocity components at key points For the first Frame number Vertical velocity components at key points and These are the normalized relative coordinates of the previous frame; S240, Combining relative coordinates and velocity vectors, constructing the first... Frame target eigenvectors , dimension ; S250, cut length is The continuous frame sequence forms the spatiotemporal feature matrix of the input Mamba temporal state-space model. .

[0010] Based on the above, step S400 includes the following steps: S410, Edge Client Receive global model parameters sent by the central server ,in For the current federal learning round; S420 and edge clients employ a long-tail distribution sampling strategy to construct local datasets. The strategy is to retain all local main class samples and a preset ratio. (like Rare class samples of ) to mitigate catastrophic forgetting; S430, based on local dataset ,by To initialize the parameters, the Mamba time-series state-space model is... In each round of local gradient descent training (Local Epochs), the local model parameters are updated. ; S440, Update the local model parameters and the number of local samples Transmitted to the central server.

[0011] Based on the above, the improved FedAvg algorithm in step S500 is as follows: S510, Central Server Receiver Parameters uploaded by each edge client participating in training and sample size ; S520, Introducing a zero-smoothing term Calculate the first Aggregate weight of each edge client To ensure numerical stability under conditions of extreme sample imbalance: ; in, For the first The weight of each client in the global aggregation For the first The number of samples used for training by each client. The total number of clients participating in this round of training. The preset anti-zero smoothing constant has a value range of [value range missing]. to This is used to prevent numerical calculation errors caused by extremely small or zero sample size; S530, Calculate the next round of global model parameters after aggregation. : ; in, For the first Global model parameters of the wheel, For the first The client in the first The local model parameters uploaded in the round; S540, will As the first The global model parameters of the wheel are distributed to all edge clients.

[0012] Based on the above, the Mamba temporal state-space model includes a linear projection layer, multiple stacked Mamba modules, and a classification head; the Mamba module is based on a selective state-space mechanism, and its continuous-time state transition equation is: ; ; in, Hidden state Regarding time The derivative, As input features, The hidden state vector. Here is the state transition matrix. For the input control matrix, To output the projection matrix, For a moment The output of .

[0013] Based on the above, in step S700, when identifying and analyzing the high-dimensional spatiotemporal feature matrix using the local Mamba temporal state space model, the inference phase also includes a motion smoothing and debouncing step: S710, for each tracking target Maintain a length of Action label sliding window ; S720, Statistical Window Internal action categories Weighted score : ; in, Action category Total score, For sliding windows The single prediction record in the data. The action label for this prediction. The confidence level of the model for this prediction; S730, the action category with the highest calculated score. Score percentage ,in This is the sum of scores for all action categories within the window; S740, only when Exceeding the preset threshold If the action label is not updated, the currently displayed action label is updated; otherwise, the label from the previous moment is left unchanged to eliminate instantaneous changes in the recognition result.

[0014] This invention has outstanding substantive features and significant progress compared to the prior art, specifically: 1) Privacy protection and compliance: The federated learning architecture is adopted, and the model parameters are trained and uploaded only at the edge. The original video data never leaves the local machine, thus protecting data privacy and strictly complying with data security regulations. 2) Solving the data silo problem: Through the federated aggregation mechanism, the limitation of insufficient samples from each client (especially the scarcity of abnormal behavior samples) is broken, enabling the global model to integrate experience from multiple parties and significantly improving the generalization ability in heterogeneous scenarios. 3) Efficient inference at the edge: Combining the spatial extraction capabilities of YOLOv11-pose with the linear temporal modeling capabilities of Mamba, a heterogeneous cascaded network was constructed, which significantly reduced computational complexity and supported real-time behavior analysis of multiple users on low-computing-power devices. 4) Robustness and stability: Normalized relative coordinates, velocity features, and smooth anti-shaking logic at the inference end are introduced to effectively eliminate the interference of illumination, position changes, and instantaneous misjudgment on the recognition results. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the process of this invention.

[0016] Figure 2 This is a schematic diagram of the spatiotemporal feature matrix construction and normalization logic of the present invention.

[0017] Figure 3 This is a schematic diagram of the interaction process between the federated learning client and server of this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, a behavior recognition method based on federated learning and Mamba networks includes the following steps: S100, acquiring a real-time video stream within a target monitoring area, and extracting the skeleton keypoint coordinate sequence and corresponding tracking ID of the human body in each frame of the real-time video stream; S200, converting the absolute pixel coordinates of the skeleton keypoint coordinate sequence into normalized relative coordinates relative to the center point of the human body, and calculating the keypoint displacement velocity vector between adjacent frames to construct a high-dimensional spatiotemporal feature matrix; S300, constructing a federated learning system, including a central server and multiple local Mamba temporal state space models; S400, using local non-independent and identically distributed datasets to perform Mamba... The Mamba temporal state-space model is trained locally to obtain updated local model parameters, which are then uploaded to the central server. In step S500, the central server uses an improved FedAvg algorithm with a zero-prevention smoothing term to globally aggregate all uploaded local model parameters, generating global model parameters, which are then distributed to each edge client for updates. In step S600, steps S400-S500 are repeated until the set number of communication rounds or global model convergence is achieved, thus obtaining a stable federated learning system. In step S700, the high-dimensional spatiotemporal feature matrix is ​​identified and analyzed using the local Mamba temporal state-space model, and the behavior recognition results are output.

[0020] The Mamba temporal state-space model includes a linear projection layer, multiple stacked Mamba modules, and a classification head. The Mamba module is based on a selective state-space mechanism, and its continuous-time state transition equation is: ; ; in: Hidden state Regarding time The derivative, As input features, The hidden state vector. Here is the state transition matrix. For the input control matrix, To output the projection matrix, For a moment The output of the equation is obtained by converting it into a discrete-time recursive form using zero-order hold (ZOH) discretization for deep learning training. Compared to traditional Transformer models, the Mamba model has linear inference complexity and can handle long sequence skeleton data more efficiently.

[0021] The specific examples are illustrated below: Implementation environment: This embodiment simulates a multi-client environment on a server equipped with an NVIDIA RTX A5000 graphics card. The software environment is based on Python 3.10, the PyTorch deep learning framework, and the Flower federated learning framework. This embodiment specifically uses a smart classroom as a monitoring scenario. Dataset construction and partitioning: The dataset used is a locally created classroom student action dataset. This dataset contains 5 action categories: normal (normal class actions) 36 video data, raise_hand (raising hands) 39 video data, sleep (sleeping actions) 29 video data, phone (using mobile phones) 68 video data, and reading (reading) 43 video data. 10% of the entire dataset is selected as the global test set to objectively evaluate the generalization ability of each model; the remaining 90% is used as the training set and partitioned into different edge clients according to the pathological non-independent identically distributed (IID) strategy to simulate the "data silo" phenomenon in the real world.

[0022] The implementation process of this invention includes the following steps M1 to M5: Step M1: Local Feature Perception and Extraction Each teaching terminal acts as a federated learning edge client, capturing real-time video feeds from classroom surveillance cameras. The system obtains raw video frames. The data is then fed into a lightweight YOLOv11n-pose model that integrates a multi-object tracking module for inference on each frame. To achieve continuous motion analysis, this embodiment enables YOLO's Track Mode, using the BoT-SORT algorithm to assign a unique tracking ID to each student in the frame. For each ID, the model outputs the bounding box of that object. (in The coordinates of the top left corner (Width and height), and coordinates containing 17 key points. and confidence level.

[0023] Step M2: Heterogeneous Temporal Feature Modeling Combination Figure 1 S200 steps and Figure 2As shown in the feature transformation pipeline: To eliminate differences in human body scale caused by varying distances between different people and the camera, this embodiment normalizes the extracted skeleton data. For example... Figure 2 As shown, the specific calculation logic is as follows: 1. Calculate the center point: For the th For a target in a frame, first calculate the coordinates of its center point based on its bounding box, using the following formula: and .

[0024] 2. Coordinate normalization: Normalize the absolute coordinates of key points. Convert to relative coordinates relative to the center point The formula is and .

[0025] 3. Velocity Feature Calculation: To capture dynamic features (distinguishing between "stationary arm-hugging" and "dynamic hand-raising"), the displacement velocity of adjacent frames is calculated using the following formula: and .

[0026] 4. Feature concatenation and temporal stacking: The normalized coordinates and velocity features mentioned above are concatenated to form a single-frame feature vector. Its dimensions are Dimension. 30 consecutive frames of data are captured, and the dimension is constructed as follows: High-dimensional spatiotemporal feature matrix Finally, it is input into the Mamba temporal state-space model.

[0027] Step M3: Cross-domain privacy collaborative training Combination Figure 1 S300-S400 steps and Figure 3 As shown in "Phase 1" and "Phase 2," this embodiment constructs a federated learning system comprising client A, client B, and a central server. This addresses the long-tail distribution of data in real-world scenarios. Client A: Represents an "actively interactive" classroom, loading a local training subset. Its data is extremely skewed, mainly consisting of samples of "normal", "raising hand" and "playing on mobile phone" (accounting for about 80%), with only a very small number of rare samples of "sleeping" and "reading" (accounting for about 20%).

[0028] Client B: Represents a "quiet study" type classroom, loading a local training subset. It mainly contains samples of "normal", "sleeping" and "reading" (accounting for about 80%), with only a very small number of rare samples of "raising hand" and "playing on mobile phone" (accounting for about 20%).

[0029] exist Figure 3In the iterative process, the central server first issues global parameters. After receiving the data, clients A and B... To initialize the parameters, gradient descent is performed on the Mamba time-series state-space model using the local dataset to update the parameters locally. and .

[0030] Step M4: Aggregation of stability enhancement parameters Combination Figure 1 S500 steps and Figure 3 As shown in "Phase 3": Each client will send the trained model parameters ( ) and the corresponding local sample size ( The data is uploaded to the central server. The server uses an improved FedAvg algorithm with a zero-smoothing term for global aggregation. For example... Figure 3 As shown in the server-side logic, to prevent the weight denominator from approaching zero due to scarce categories, a zero-division smoothing term is introduced when calculating the weights. That is, aggregate weight Using this weight, a new global model is calculated. and will Reissued to all clients.

[0031] Step M5: Model Convergence and Inference Debouncing Combination Figure 1 Mid-step S600-700 and actual system deployment logic: The system repeats the execution Figure 3 The S400-S500 loop, after 20 rounds of communication iterations, led to global model convergence. The average accuracy on the test set exceeded 95%, with no weaknesses in either the "raise hand" or "sleep" categories. In the practical application inference phase, to address transient transitions, the system introduced the ActionSmoother module: maintaining a 15-frame sliding window of action labels for each tracked target. Only when a certain action (such as "sleeping") is within the window is the weighted score percentage determined. Exceeding the preset threshold (like Only when the action label is updated is the final action label displayed updated, which effectively ensures the smoothness and stability of behavior recognition.

[0032] Experimental Comparison and Demonstration: To verify the effectiveness of this invention in breaking down "data silos" while protecting privacy, isolated local models (Client A, Client B) and federated global models were evaluated on a unified 10% global test set. The test results are shown in Table 1.

[0033] Table 1. Accuracy comparison between isolated and federated models on the global test set. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A behavior recognition method based on federated learning and Mamba networks, characterized in that, Including the following steps: S100. Obtain the real-time video stream within the target monitoring area, and extract the coordinate sequence of the human skeleton key points and the corresponding tracking ID in each frame of the real-time video stream. S200. Convert the absolute pixel coordinates of the skeleton key point coordinate sequence into normalized relative coordinates relative to the human body center point, and calculate the key point displacement velocity vector between adjacent frames to construct a high-dimensional spatiotemporal feature matrix. S300: Construct a federated learning system, including a central server and multiple local Mamba time-series state-space models; S400. Use the local non-independent and identically distributed dataset to train the Mamba time-series state-space model locally, obtain the local model parameter update, and upload the updated local model parameters to the central server. S500: The central server performs global aggregation on all uploaded local model parameters, generates global model parameters, and distributes them to each edge client for updates. S600, repeat steps S400-S500 until the set number of communication rounds or global model convergence is reached, thereby obtaining a stable federated learning system; S700: The high-dimensional spatiotemporal feature matrix is ​​identified and analyzed using the local Mamba temporal state space model, and the behavior recognition results are output.

2. The behavior recognition method based on federated learning and Mamba networks according to claim 1, characterized in that, Step S100 includes the following steps: S110, Input the... Frame video image Multi-scale image features were extracted using the YOLOv11 backbone network. S120, Outputs human body bounding box via detection head. and confidence level ,in For the detected target index, The coordinates of the top left corner of the bounding box. These are the width and height of the bounding box, respectively; S130. Use the BoT-SORT tracking algorithm to associate targets in adjacent frames and assign a unique tracking ID to each target. , thus obtaining the trajectory set ; S140, Regressing to Each Target Through Key Nods of Coordinates of key points and visibility confidence ,in , , For the first Frame number The x-coordinates of the key points For the first Frame number The ordinates of the key points.

3. The behavior recognition method based on federated learning and Mamba networks according to claim 1, characterized in that, Step S200 includes the following steps: S210, Calculation Target In the Coordinates of the human body center point in the frame image : ； in, For the goal The x-coordinate of the center point, For the goal The ordinate of the center point, For the goal The x-coordinate of the top-left corner of the bounding box. For the goal The y-coordinate of the top-left corner of the bounding box. For the goal The width of the bounding box, For the goal The height of the bounding box; S220, Set the absolute coordinates of key points Convert to normalized relative coordinates To eliminate the influence of the human body's position and scale in the image: ； in, For the first Frame number Normalized relative x-coordinates of key points For the first Frame number Normalized relative ordinates of key points The x-coordinate of the original pixel. The original pixel ordinate; S230. Calculate the normalized velocity vector of key points between adjacent frames. To capture the dynamic characteristics of the action: ； in, For the first Frame number The horizontal velocity components at key points For the first Frame number Vertical velocity components at key points and These are the normalized relative coordinates of the previous frame; S240, Combining relative coordinates and velocity vectors, constructing the first... Frame target eigenvectors , dimension ; S250, cut length is The continuous frame sequence forms the spatiotemporal feature matrix of the input Mamba temporal state-space model. .

4. The behavior recognition method based on federated learning and Mamba networks according to claim 1, characterized in that, Step S400 includes the following steps: S410, Edge Client Receive global model parameters sent by the central server ,in For the current federal learning round; S420 and edge clients employ a long-tail distribution sampling strategy to construct local datasets. The strategy is to retain all local main class samples and a preset ratio. Rare class samples to mitigate catastrophic forgetting; S430, based on local dataset ,by To initialize the parameters, the Mamba time-series state-space model is... During the local gradient descent training round, the local model parameters are updated to... ; S440, Update the local model parameters and the number of local samples Transmitted to the central server.

5. The behavior recognition method based on federated learning and Mamba networks according to claim 1, characterized in that, Step S500 includes the following steps: S510, Central Server Receiver Parameters uploaded by each edge client participating in training and sample size ; S520, Introducing a zero-smoothing term Calculate the first Aggregate weight of each edge client To ensure numerical stability under conditions of extreme sample imbalance: ； in, For the first The weight of each client in the global aggregation For the first The number of samples used for training by each client. The total number of clients participating in this round of training. The preset anti-zero smoothing constant has a value range of [value range missing]. to This is used to prevent numerical calculation errors caused by extremely small or zero sample size; S530, Calculate the next round of global model parameters after aggregation. : ； in, For the first Global model parameters of the wheel, For the first The client in the first The local model parameters uploaded in the round; S540, will As the first The global model parameters of the wheel are distributed to all edge clients.

6. The behavior recognition method based on federated learning and Mamba networks according to claim 1, characterized in that: The Mamba temporal state-space model includes a linear projection layer, multiple stacked Mamba modules, and a classification head; the Mamba module is based on a selective state-space mechanism, and its continuous-time state transition equation is: ； ； in, Hidden state Regarding time The derivative of For input features, The hidden state vector. Here is the state transition matrix. For the input control matrix, To output the projection matrix, For a moment The output.

7. The behavior recognition method based on federated learning and Mamba networks according to claim 1, characterized in that: In step S700, when identifying and analyzing the high-dimensional spatiotemporal feature matrix using the local Mamba temporal state-space model, the inference phase also includes a motion smoothing and debouncing step: S710, for each tracking target Maintain a length of Action label sliding window ; S720, Statistical Window Internal action categories Weighted score : ； in, Action category Total score, For sliding windows The single prediction record in the data. The action label for this prediction. The confidence level of the model for this prediction; S730, the action category with the highest calculated score. Score percentage ,in The sum of scores for all action categories within the window; S740, only when Exceeding the preset threshold If the action label is not updated, the currently displayed action label is updated; otherwise, the label from the previous moment is left unchanged to eliminate instantaneous changes in the recognition result.

Images