A video behavior recognition method under personalized federated learning

Abstract

The application discloses a video behavior recognition method under personalized federated learning, first constructs a non-independent and identically distributed video behavior recognition data set to simulate a federated learning scene; then a video behavior recognition model based on space-time features is built, mainly including a local module and a global module, the local module realizes extraction of space-time features on video segmentation; the global module extracts motion rhythm features by using differential information at the feature level, and realizes deep extraction and fusion of space-time information on the whole video; then training is carried out under the condition of federated learning, each user completes training in parallel based on a local data set, local module parameters are taken as local private parameters, and global module parameters are returned to a server, the server aggregates parameters returned by each user to obtain shared parameters, and the shared parameters are transmitted to the user, and the user realizes video behavior classification by relying on the local private parameters and the shared parameters.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video behavior recognition, and more specifically relates to a video behavior recognition method under personalized federated learning. BACKGROUND

[0002] As an important research direction of artificial intelligence, video behavior recognition for video data has developed rapidly relying on deep learning algorithms. Video data contains spatial and temporal features, and common research methods rely on complex optical flow and three-dimensional convolution to realize spatio-temporal feature extraction. The method based on complex optical flow extracts spatial and temporal features through a double-flow structure, one flow uses RGB frames to extract spatial features, and the other flow uses optical flow to extract temporal features, and the video behavior recognition result is obtained by combining the class scores of the two flows. However, the extraction of optical flow information relies on a separate model or tool, which requires complex preprocessing and a larger model to realize feature extraction, which is not conducive to research and application. At the same time, the double-flow network often extracts features based on a segment of video data, which can only obtain local features and cannot fully utilize video data. The method based on three-dimensional convolution expands the convolution kernel to three dimensions, and directly extracts spatio-temporal features based on video data, but the expansion of the convolution kernel dimension also greatly increases the parameter quantity.

[0003] Considering the accuracy and efficiency of the model, we use differential technology to realize the extraction of time information, and propose a new differential information extraction method and an action rhythm feature extraction method, which uses simple and efficient differential technology to realize the extraction of time and space features in local and global ranges respectively.

[0004] Secondly, considering the user privacy and data security issues, it is not convenient to upload the user's local video data to the server for model training, and combined with the federated learning algorithm, the local training and center aggregation of the model can be realized. However, the existing federated learning method is often based on natural language processing or image recognition data sets for research, which has low matching degree with video behavior recognition models and data sets. We combine the characteristics of video behavior recognition models and propose a personalized federated learning scheme suitable for video models. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a video behavior recognition method under personalized federated learning, which improves the local recognition effect under federated learning while considering user privacy and data security issues.

[0006] To achieve the above-mentioned purpose of the application, the video behavior recognition method under personalized federated learning comprises the following steps:

[0007] (1) making a federated learning video data set;

[0008] Download a publicly available video dataset containing N behavior categories;

[0009] The total number of users participating in federated learning training is set to C. The public video dataset is divided into C subsets using the Dirichlet method, so that each user receives a subset, and each subset contains data of N categories. The subsets are non-independent and identically distributed.

[0010] (2) Construct a video behavior recognition model based on spatiotemporal features;

[0011] The video behavior recognition model based on spatiotemporal features includes an input module, a local feature extraction module, multiple global feature extraction modules in series, as well as a pooling layer and a classifier;

[0012] The input module segments the input data, then randomly samples k consecutive frames of images from each segment, and then inputs the n sampled images into the local feature extraction module, where k is an odd number.

[0013] The local feature extraction module comprises two network branches. The first branch first performs frame-level difference calculations on each segment of k frames, using the intermediate frame as the reference, followed by channel-level smoothing. Then, it sequentially performs average pooling and inputs the data through pooling layers to a convolutional layer. After convolution, it performs upsampling to obtain a sampled output. This sampled output is then subjected to convolution again by the convolutional layer to obtain supplementary temporal features. The second branch directly performs convolution on the intermediate frame of each segment to obtain a convolutional output. This convolutional output is added to the sampled output from the first branch and then subjected to convolution again to obtain local spatial features. Finally, the supplementary temporal features and local spatial features are added together and passed through a pooling layer to obtain the local features.

[0014] In each global feature extraction module, the input local features are first compressed in the channel dimension, and then the forward feature difference and backward feature difference are obtained through feature-level difference calculation.

[0015] The forward feature difference is used as the input to three branches. The first branch is fed into convolutional layer 3 after passing through convolutional layer 1, pooling layer, and upsampling layer. The second branch is fed into convolutional layer 3 after passing through convolutional layer 2. The third branch is directly fed into convolutional layer 3. In convolutional layer 3, the outputs of the three branches are added together and then convolved again before being activated by the activation layer to obtain the initial forward difference information.

[0016] The backward feature difference process is the same as the forward feature difference process, so the initial backward difference information can be obtained in the same way.

[0017] In the attention mechanism module, the initial difference information of the two sides is added together and multiplied with the local features of the input one by one. After being weighted by the attention mechanism, it is added again with the local features of the input. Then, the added structure is convolved through the convolution layer to obtain the output of the global module.

[0018] Finally, the global features are processed through pooling layers and a classifier to output the prediction results;

[0019] (3) Train a video behavior recognition model based on spatiotemporal features;

[0020] (3.1) Let S be the number of users participating in federated learning in each round, that is, S / C data are trained in parallel in each round; at the beginning of each round of training, S users are randomly sampled from the total number of users C, and then the subset of data corresponding to these S users is used as training data.

[0021] (3.2) Train S recognition models in parallel using S users;

[0022] In the training process of each recognition model, the video data in each subset is first divided into n segments by the input module, and k consecutive frames of images are randomly sampled in each segment. Then, the n sampled images are input into the local feature extraction module.

[0023] In the local feature extraction module, local features F of each video segment are extracted. i , i = 1, 2, ..., n;

[0024] In the cascaded full-local feature extraction module, the local features F of each video segment are... i The input is fed into multiple serially connected global and local feature extraction modules, and the final global features are obtained through repeated feature extraction.

[0025] Finally, the global spatiotemporal features are passed through a pooling layer and input into a classifier to predict the behavior category of each video segment;

[0026] The cross-loss value is calculated by comparing the video behavior category predicted by the recognition model with the corresponding real video behavior category. Then, it is determined whether the current iteration number has reached a preset value or whether the cross-loss value is less than a preset threshold. If the above conditions are met, the iterative training is stopped and the trained video behavior recognition model is obtained. Otherwise, the cross-loss value is backpropagated and the weight parameters of the video behavior recognition model are updated by the gradient descent algorithm, and then the next round of training is carried out.

[0027] (4) User-local implementation of video behavior recognition and classification;

[0028] After personalized federated learning is completed, each user saves a video behavior recognition model locally. When performing video behavior recognition and classification tasks, users collect video data locally in real time and input it into the local video behavior recognition model, thereby achieving local prediction of video behavior.

[0029] The objective of this invention is achieved as follows:

[0030] This invention discloses a personalized federated learning method for video behavior recognition. First, a non-independent, identically distributed video behavior recognition dataset is constructed to simulate a federated learning scenario. Then, a video behavior recognition model based on spatiotemporal features is built, mainly comprising local and global modules. The local module extracts spatiotemporal features at the video segmentation level; the global module extracts action rhythm features using feature-level differential information, achieving deep extraction and fusion of spatiotemporal information across the entire video. Training is then conducted under federated learning conditions, with each user completing parallel training based on their local dataset. The parameters of the local modules are used as local private parameters, while the parameters of the global modules are sent back to the server. The server aggregates the parameters sent back by each user to obtain shared parameters, which are then transmitted back to the users. Users then use their local private parameters and shared parameters to classify video behaviors.

[0031] Meanwhile, the video behavior recognition method under personalized federated learning of this invention also has the following beneficial effects:

[0032] (1) A new differential information extraction scheme is proposed for the local spatiotemporal features of video segments. Differential features are extracted with single frame RGB as the center, which improves the local feature extraction effect.

[0033] (2) To address the global temporal features of a complete video, a method for extracting action rhythm features using differential techniques is proposed, which improves the global feature extraction effect.

[0034] (3) For video behavior recognition under federated learning conditions, a personalized federated learning scheme suitable for video models is proposed. Based on the structure of the video behavior recognition model, a new personalized federated learning training scheme is proposed, which improves the model training effect. Attached Figure Description

[0035] Figure 1 This is a flowchart of a video behavior recognition method based on personalized federated learning according to the present invention;

[0036] Figure 2 It is based on the principle of video behavior recognition model with spatiotemporal features;

[0037] Figure 3 This is a flowchart of the processing of each segment of data. Detailed Implementation

[0038] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0039] Example

[0040] Figure 1 This is a flowchart of a video behavior recognition method based on personalized federated learning according to the present invention.

[0041] In this embodiment, as Figure 1 As shown, the present invention provides a video behavior recognition method based on personalized federated learning, comprising the following steps:

[0042] S1. Create a federated learning video dataset;

[0043] Download the publicly available video dataset UCF-101, which contains 101 categories of video behavior data;

[0044] The total number of users participating in federated learning training is set to 20. The public video dataset is divided into 20 subsets using the Dirichlet method, so that each user gets one subset. Each subset contains data of 101 categories, but the number of data in each category varies greatly in different subsets. The subsets follow a non-independent identically distributed distribution to simulate the distribution of user data in a federated learning scenario.

[0045] S2. Construct a video behavior recognition model based on spatiotemporal features;

[0046] like Figure 2 As shown, the video behavior recognition model based on spatiotemporal features includes an input module, a local feature extraction module, three cascaded global feature extraction modules, a pooling layer, and a classifier.

[0047] The input module segments the input data, then randomly samples 5 consecutive frames of images from each segment, and then inputs the 8 sampled images into the local feature extraction module.

[0048] The local feature extraction module comprises two network branches. The first branch first performs frame-level difference calculation based on the middle frame of each 5-frame segment, then performs channel-level smoothing, followed by average pooling and inputting the data through pooling layers into a convolutional layer. After convolution, upsampling is performed to obtain the sampled output. The sampled output is then upsampled again after convolution in the convolutional layer to obtain supplementary temporal features. The second branch directly performs convolution on the middle frame of each segment to obtain the convolutional output. The convolutional output is added to the sampled output from the first branch and then convolutioned again to obtain local spatial features. Finally, the supplementary temporal features and local spatial features are added together and passed through a pooling layer to obtain the local features.

[0049] In each global feature extraction module, the input local features are first compressed in the channel dimension, and then the forward feature difference and backward feature difference are obtained through feature-level difference calculation.

[0050] The forward feature difference is used as the input to three branches. The first branch passes through a convolutional layer, a pooling layer, an upsampling layer, and then is input to a convolutional layer. The second branch passes through a convolutional layer and then is input to another convolutional layer. The third branch is directly input to the convolutional layer. In the convolutional layer, the outputs of the three branches are added together, and then a convolution operation is performed again before activation by an activation layer to obtain the initial forward difference information.

[0051] The backward feature difference process is the same as the forward feature difference process, so the initial backward difference information can be obtained in the same way.

[0052] In the attention mechanism module, the initial difference information from both directions is added together and multiplied one by one with the local features of the input. After being weighted by the attention mechanism, it is added again with the local features of the input. The added structure is then convolved through a convolutional layer to obtain the global features.

[0053] Finally, the global features are processed through pooling layers and a classifier to output the prediction results;

[0054] S3. Train a video behavior recognition model based on spatiotemporal features;

[0055] S3.1 Assume that the number of users participating in federated learning in each round is 4, that is, 1 / 5 of the data is trained in parallel in each round; at the beginning of each round of training, 4 users are randomly sampled from the total number of users of 20, and then the subset of data corresponding to these 4 users is used as training data;

[0056] S3.2. Four video behavior recognition models are trained in parallel by four users;

[0057] In each recognition model training process, such as Figure 3As shown, the video data in each subset is first divided into 8 segments by the input module, and 5 consecutive frames are randomly sampled from each segment. Then, the 8 sampled frames are input into the local feature extraction module. In this embodiment, the 5 sampled frames of each segment are denoted as I. t-2 ,I t-1 ,I t ,I t+1 ,I t+2 The intermediate frame is I t t represents the sampling position of the frame;

[0058] In the local feature extraction module, the first branch extracts the I of each segment. t-2 ,I t-1 I t+1 ,I t+2 These four frames of data are respectively compared with the intermediate frame I. t Perform difference calculations to obtain I t The first branch performs frame-level difference analysis on the 4 core frames, followed by channel-level smoothing. Then, it sequentially performs average pooling and inputs the data through pooling layers to a convolutional layer. After convolution, it is upsampled to obtain the sampled output. This sampled output is then upsampled again after convolution in the convolutional layer to obtain supplementary temporal features. The second branch directly extracts the intermediate frames I from each segment. t The convolutional output is obtained through convolutional operations in the convolutional layer. This output is then added to the sampled output from the first branch and passed through another convolutional layer to obtain local spatial features. Finally, the supplementary temporal features are added to the local spatial features and then passed through a pooling layer to extract the local features F of each video segment. i , i = 1, 2, ..., n;

[0059] In the global feature extraction stage, local features F i The input is fed into the first global feature extraction module. After channel-dimensional compression, the local features of all segments are concatenated in chronological order to obtain the local feature vector F = [F1, F2, ..., F]. i ,…,F n The local feature vectors are then subjected to feature-level difference calculations to obtain the forward feature difference (Diff). 前向 Difference with backward features Diff 后向 ;

[0060] Among them, forward feature difference Diff 前向 Difference with backward features Diff 后向 The calculation process is as follows:

[0061] First calculate the forward feature F 前向 With backward feature F 后向 :

[0062] In this embodiment, F前向 and F 后向 Each step is taken as a reference point and shifted one segment forward and backward. The excess portion is removed, and blank portions are filled with 0s to obtain F. 前向 =[F2,F3,…,F n ,0],F 后向 =[0,F1,…,F n-2 ,F n-1 ]; then F 前向 and F 后向 By performing difference calculations with F separately, we obtain the bidirectional characteristic difference:

[0063] Diff 前向 =[F2-F1,F3-F2,…,F n -F n-1 ,-F N ]

[0064] Diff 后向 = [-F1, F1-F2, ..., F n-2 -F n-1 ,F n-1 -F n ]

[0065] Forward Feature Difference (Diff) 前向 Difference with backward features Diff 后向 The subsequent processing flow is exactly the same. Below, we will discuss the forward feature difference Diff. 前向 For example, the following is a detailed explanation:

[0066] Forward Feature Difference (Diff) 前向 As input to three branches, the first branch passes through convolutional layer 1, a pooling layer, and an upsampling layer before being input to convolutional layer 3; the second branch passes through convolutional layer 2 before being input to convolutional layer 3; and the third branch is directly input to convolutional layer 3. In convolutional layer 3, the outputs of the three branches are summed, convolved again, and then activated by an activation layer to obtain the initial forward difference information F. D前向 ;

[0067] Similarly, the initial backward difference information F can be obtained. D后向 ;

[0068] In the attention mechanism module, the global difference feature F is obtained by adding the initial difference information from both directions. D F D Then, it is multiplied with the local feature vector F one by one, and after being weighted by the attention mechanism, it is added back to the local feature vector F. Then, the added structure is convolved through a convolutional layer to obtain the global features.

[0069] The global features extracted by the previous global feature extraction module are used as the input for the next global feature extraction module. This process is repeated until the global features F of the global feature extraction stage are obtained. 全局 ;

[0070] Finally, the global feature F is... 全局 By using a pooling layer and inputting it into a classifier, the behavior category of each video segment can be predicted.

[0071] The cross-loss value is calculated by comparing the video behavior category predicted by the recognition model with the corresponding real video behavior category. Then, it is determined whether the current iteration number has reached a preset value or whether the cross-loss value is less than a preset threshold. If the above conditions are met, the iterative training is stopped and the trained recognition model is obtained. Otherwise, the cross-loss value is backpropagated and the weight parameters of the recognition model are updated by the gradient descent algorithm, and then the next round of training is carried out.

[0072] S4. Users can implement video behavior recognition and classification locally;

[0073] After personalized federated learning is completed, each user saves a video behavior recognition model locally. When performing video behavior recognition and classification tasks, users collect video data locally in real time and input it into the local video behavior recognition model, thereby achieving local prediction of video behavior.

[0074] Experiment and Results Analysis

[0075] We first conducted experiments to validate multiple optional parameters and schemes in the video behavior recognition model, compared the model performance under multiple metrics, and presented the recognition accuracy on the UCF-101 dataset. Furthermore, based on the non-independent and identically distributed UCF-101 dataset, we verified the effectiveness of personalized federated learning and tested the experimental results under multiple federated learning hyperparameters.

[0076] Experimental setup

[0077] All our models are built on the ResNet-50 architecture. The operating system is Ubuntu 22.04.1LTS, with an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. GPUs were used for model training and testing, with an NVIDIA GeForce RTX 3090Ti graphics card and 24GB of RAM. Our models are built using the PyTorch framework; the experimental environment is Python 3.7.15 + PyTorch 1.10 + CUDA 11.3.

[0078] Regarding training parameters, for RGB data of size 256×320, we randomly cropped it to 224×224 when importing the dataset, and the number of classes in the dataset is 101. The default number of segments in the experiment was 8, meaning each video data was divided into 8 segments, and 5 consecutive RGB images were sampled from each segment. The local information of each video segment in the model is mainly based on one RGB image frame, but because RGB difference needs to be calculated, 5 RGB images are used in each segment, for a total of 5×8=40 RGB images per video data.

[0079] Regarding the setting and adjustment of the learning rate, in our comparative experiment, we set the learning rate decay step size to [40, 60, 70] and the decay factor to 0.1, that is, at the 40th, 60th, and 70th rounds, the learning rate was adjusted to 0.1 times the current value, and the dropout was set to 0.5.

[0080] Differential RGB extraction method

[0081] In the spatial module based on local information, this section presents a comparative experiment to test the performance of each of the three differential RGB information extraction schemes.

[0082] Table 1. Recognition results under different differential feature extraction methods;

[0083]

[0084] Table 1

[0085] Table 1, I t ,t∈{1,2,3,4,5} represents the RGB frames at time t, I3 is obtained by random sampling and is used for spatial feature extraction, and the other 4 frames are 2 frames before and after time I3, Diff i-j Representing frame I i With frame I j The difference information was extracted. To demonstrate the effectiveness of differential RGB, the model performance without differential information was first tested by directly superimposing the two frames before and after the sampled frame using the Concat function. Experimental results show that local modules using differential information achieve better experimental results.

[0086] Regarding the specific method of obtaining differential information, directly replacing the difference between adjacent frames with the difference between the sampled frame actually reduces the accuracy. Considering that the greater the time distance between RGB frames, the greater the difference, and the more noise the differential information will contain, this paper first uses an average pooling layer in the channel dimension after obtaining the differential frames to make the differential frames smoother in the channel dimension and appropriately compress the difference between differential features at different times. Then, average pooling and channel pooling operations are performed on each frame separately, and the pooled features are superimposed to achieve better experimental results. On the UCF-101 dataset, the final accuracy reached 85.851%.

[0087] Translation stride

[0088] In the global feature module, experimental results were tested under different translation step schemes.

[0089] Table 2 presents the Top 1 and Top 5 accuracies achieved with different translation strides when performing local feature differencing in the global module. The experimental results show that the model with a stride of 1-1-2 achieves a higher Top 1 accuracy, improving accuracy by 0.487% compared to the original 1-1-1 approach. The model with a stride of 1-2-2 achieves a 0.027% improvement in Top 5 accuracy compared to the original approach, validating the effectiveness of global phased differencing features on the UCF-101 dataset. However, when the stride is set to 1-2-3, the recognition accuracy decreases significantly, indicating that differencing information with a larger time span is no longer effective and may even negatively impact recognition performance.

[0090] Serial number Step size Accuracy Top1 Accuracy Top5 1 1-1-1 85.444％ 97.132％ 2 2-2-2 85.038％ 96.943％ 3 1-1-2 85.931％ 97.051％ 4 1-2-1 85.092％ 96.997％ 5 1-2-2 85.363％ 97.159％ 6 1-2-3 84.686％ 96.510％

[0091] Table 2

[0092] Learning rate

[0093] The learning rate training strategy varies depending on the model and dataset, which affects whether parameter updates can reach the optimal point and whether the model can achieve the highest accuracy. Our model was trained on the UCF-101 dataset, and we conducted extensive experimental evaluations based on different initial learning rates and learning rate decay steps.

[0094] First, experiments were conducted with different initial learning rate settings. The initial learning rate was selected based on commonly used initial learning rates in the field. The decay step size was uniformly set to [30, 45, 55], and the decay factor was 10. A total of 60 training rounds were conducted.

[0095] Serial number Initial learning rate Accuracy Top1 Epoch 1 0.01 84.172％ 57 2 0.02 80.628％ 41 3 0.001 86.120％ 40 4 0.002 85.931％ 46

[0096] Table 3

[0097] Table 3 shows the top-performing accuracy of our model and the corresponding training epochs when the initial learning rate is set to 0.01, 0.02, 0.001, and 0.002. The experimental results show that the model achieves optimal training performance when the initial learning rate is set to 0.001, with a recognition accuracy of 86.12% in the 40th training epoch. Therefore, we set the initial learning rate to 0.001.

[0098] Next, we conducted tests based on different decay step sizes. Based on the previous experimental results, we set the initial learning rate to 0.001 and the total number of training rounds to 80. We then added 20 more rounds to better analyze the experimental effects of multi-round training. The experimental results are shown in Table 4.

[0099] Serial number Decay step Accuracy Top1 Epoch 1 20,25,30 85.741％ 28 2 20,40,55 85.741％ 40 3 30,45,55 86.120％ 40 4 40,60,70 86.959％ 60

[0100] Table 4

[0101] As can be seen from the experimental results in Table 4, when the decay step size is set to [40, 60, 70], the model achieves the best training effect, with an accuracy of 86.959% in the test results after the 60th round of training.

[0102] Model test results under optimal parameters

[0103] Finally, based on the optimal model scheme and hyperparameters obtained from the above experiments, we present the best recognition accuracy based on the UCF-101 dataset. Considering the input data scale of other models, the number of data segments N = 16 is set in the experiments in Table 5.

[0104]

[0105] Table 5

[0106] Table 5 shows the accuracy comparison between our model and other action recognition models. The TSN and C3D models were pre-trained only on simple datasets like ImageNet or Sports-1M. Even with this simple pre-training, our model achieved the highest accuracy of 87%. The StNet, TSM, STM, and TDN models were further pre-trained on the large-scale Kinetics-400 dataset. Because the number of samples in this dataset is much larger than in UCF-101, it can learn more complex data representations through training, thus significantly improving the final accuracy. Again, under the ImageNet+Kinetics pre-training model conditions, our model still achieved the highest recognition accuracy of 97.6%.

[0107] Personalized Federated Learning Effects

[0108] In the experiment, the total number of users was set to 20, the number of user samples in each federated learning communication round was set to 4, and the number of local training rounds for each user was set to 1. The dataset was grouped using Dir(1). Under the above parameter settings, the proposed video behavior recognition optimization model based on personalized federated learning was experimentally verified, and the experimental results were compared under the two conditions of conventional federated learning training and personalized federated learning.

[0109] Table 6 shows the top 1 and top 5 accuracies achieved by our model on a local dataset of 20 users under both conventional and personalized federated learning. The average accuracy shows that our proposed personalized federated learning scheme achieves better results on both metrics, improving by 1.55% on Top 1 and 0.079% on Top 5, validating the effectiveness of the personalized federated learning method for video behavior recognition.

[0110]

[0111]

[0112] Table 6

[0113] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A video behavior recognition method based on personalized federated learning, characterized in that, Includes the following steps: (1) Create a federated learning video dataset; Download a publicly available video dataset containing N behavior categories; The total number of users participating in federated learning training is set to C. The public video dataset is divided into C subsets using the Dirichlet method, so that each user receives a subset, and each subset contains data of N categories. The subsets are non-independent and identically distributed. (2) Construct a video behavior recognition model based on spatiotemporal features; The video behavior recognition model based on spatiotemporal features includes an input module, a local feature extraction module, multiple global feature extraction modules in series, as well as a pooling layer and a classifier; The input module segments the input data, then randomly samples k consecutive frames of images from each segment, and then inputs the n sampled images into the local feature extraction module, where k is an odd number. The local feature extraction module comprises two network branches. The first branch first performs frame-level difference calculations on each segment of k frames, using the intermediate frame as the reference, followed by channel-level smoothing. Then, it sequentially performs average pooling and inputs the data through pooling layers to a convolutional layer. After convolution, it performs upsampling to obtain a sampled output. This sampled output is then subjected to convolution again by the convolutional layer to obtain supplementary temporal features. The second branch directly performs convolution on the intermediate frame of each segment to obtain a convolutional output. This convolutional output is added to the sampled output from the first branch and then subjected to convolution again to obtain local spatial features. Finally, the supplementary temporal features and local spatial features are added together and passed through a pooling layer to obtain the local features. In each global feature extraction module, the input local features are first compressed in the channel dimension, and then the forward feature difference and backward feature difference are obtained through feature-level difference calculation. The forward feature difference is used as the input to three branches. The first branch is fed into convolutional layer 3 after passing through convolutional layer 1, pooling layer, and upsampling layer. The second branch is fed into convolutional layer 3 after passing through convolutional layer 2. The third branch is directly fed into convolutional layer 3. In convolutional layer 3, the outputs of the three branches are added together and then convolved again before being activated by the activation layer to obtain the initial forward difference information. The backward feature difference process is the same as the forward feature difference process, so the initial backward difference information can be obtained in the same way. In the attention mechanism module, the initial difference information of the two sides is added together and multiplied with the local features of the input one by one. After being weighted by the attention mechanism, it is added again with the local features of the input. Then, the added structure is convolved through the convolution layer to obtain the output of the global module. Finally, the global features are processed through pooling layers and a classifier to output the prediction results; (3) Train a video behavior recognition model based on spatiotemporal features; (3.1) Let S be the number of users participating in federated learning in each round, that is, S / C data are trained in parallel in each round; at the beginning of each round of training, S users are randomly sampled from the total number of users C, and then the subset of data corresponding to these S users is used as training data. (3.2) Train S recognition models in parallel using S users; In the training process of each recognition model, the video data in each subset is first divided into n segments by the input module, and k consecutive frames of images are randomly sampled in each segment. Then, the n sampled images are input into the local feature extraction module. In the local feature extraction module, local features F of each video segment are extracted. i , i = 1, 2, ..., n; In the cascaded full-local feature extraction module, the local features F of each video segment are... i The input is fed into multiple serially connected global and local feature extraction modules, and the final global features are obtained through repeated feature extraction. Finally, the global spatiotemporal features are passed through a pooling layer and input into a classifier to predict the behavior category of each video segment; The cross-loss value is calculated by comparing the video behavior category predicted by the recognition model with the corresponding real video behavior category. Then, it is determined whether the current iteration number has reached a preset value or whether the cross-loss value is less than a preset threshold. If the above conditions are met, the iterative training is stopped and the trained video behavior recognition model is obtained. Otherwise, the cross-loss value is backpropagated and the weight parameters of the video behavior recognition model are updated by the gradient descent algorithm, and then the next round of training is carried out. (4) User-local implementation of video behavior recognition and classification; After personalized federated learning is completed, each user saves a video behavior recognition model locally. When performing video behavior recognition and classification tasks, users collect video data locally in real time and input it into the local video behavior recognition model, thereby achieving local prediction of video behavior.

Images