A federated self-supervised skeleton point action recognition method for improving global model aggregation

Abstract

This invention discloses an improved federated self-supervised skeletal point action recognition method for global model aggregation. The method employs a distributed training paradigm, given K clients, each possessing its own non-independent and identically distributed privacy data. Each client's local model is trained locally using its own privacy data. The server then aggregates multiple client local models into a global model through weighted collaborative training. This invention significantly enhances the generalization ability of the local model encoder and, during global model aggregation, seeks commonalities between client local model parameters and global model parameters while preserving certain differences, thus obtaining a more comprehensive global model. This invention has broad application prospects in many fields such as intelligent monitoring and human-computer interaction.

Description

Technical Field

[0001] This invention relates to a method for implementing federated skeleton point action recognition in the field of computer vision technology, and more particularly to an improved method for implementing federated self-supervised skeleton point action recognition based on global model aggregation. Background Technology

[0002] Human action recognition is a challenging computer vision task, widely used in video surveillance, human-computer interaction, and other fields. In recent years, self-supervised action recognition based on skeletal point data has attracted considerable attention due to its robustness and efficiency. However, traditional centralized learning paradigms require sharing human skeleton sequence data for training, raising concerns about data privacy. With increasing emphasis on privacy, Federated Learning (FL) has emerged as a promising collaborative training method that can protect data privacy. Federated Learning is a distributed learning paradigm that collaboratively trains multiple client-side local models and aggregates them into a general global model on the server side. Global model aggregation is a key step in Federated Learning, allowing the system to benefit from decentralized distributed data while maintaining data privacy and security.

[0003] However, the non-independent and identically distributed nature of client data in federated learning leads to data heterogeneity, increasing the difficulty of updating the global model. When local model updates from different data sources are combined to update the global model, the global model must adapt to and reconcile these differences without overfitting to any single data source or losing its generalization ability. Some existing federated learning methods address this issue by aligning local model parameters with global model parameters, while others use contrastive learning to widen the gap between the parameters of the local model in the current iteration and those in the previous iteration. However, since data heterogeneity always exists, allowing some differences between local models and finding more commonalities between the local and global models to approximate the global model helps alleviate the data heterogeneity problem.

[0004] Secondly, after the current round of client-side model training is completed, its encoder parameters are updated to the server. Subsequently, the parameters of all client-side encoders are weighted and aggregated into a global model. Existing weighted aggregation methods primarily calculate weight coefficients based on the amount of client-side data, the training loss of the client-side model, or the divergence between the client-side model parameters and the global model parameters. In recent years, weighted aggregation methods that calculate the divergence between the parameters of the client-side model and the global model have been very effective in addressing data heterogeneity and have therefore received the most attention. However, existing methods of this type mainly seek commonalities between the client-side model parameters and the global model parameters, resulting in a less comprehensive global model.

[0005] The two issues mentioned above affect skeletal action recognition by making the global model aggregated on the server side insufficiently generalizable and comprehensive, failing to demonstrate good skeletal action recognition performance on the privacy-preserving skeletal point data of each client. Therefore, it is necessary to seek a more effective contrastive learning method to enhance the generalization of the client-side local model encoder, and a better weight aggregation method to aggregate a more comprehensive global model, in order to reduce the problems caused by data heterogeneity. Summary of the Invention

[0006] Objective: To address the shortcomings of existing technologies, this invention provides an improved federated self-supervised skeletal point action recognition method based on global model aggregation. The main improvements are as follows:

[0007] 1) To enhance the generalization of the client encoder, a client-specific projection layer regularization is added to the existing client encoder regularization. This is done by increasing the distance between the projection layer parameters of other clients and decreasing the distance between the projection layer parameters of the previous round of the same client, thereby indirectly promoting the training of a more generalized client encoder.

[0008] 2) In the process of global model aggregation, a singular value decomposition parameter divergence aggregation strategy is proposed based on the idea of ​​seeking commonalities and reserving differences. This strategy can fine-tune the parameter vector of the client encoder to keep it consistent with the parameter vector of the global model encoder, thereby reducing the divergence of parameter vectors between the client and the server.

[0009] Technical Solution: An improved federated self-supervised skeletal point action recognition method based on global model aggregation. The method employs a distributed training paradigm. The main idea is to provide K clients, each possessing its own non-independent and identically distributed privacy data (denoted by D*). Each client's local model is trained locally using its own privacy data. The server aggregates multiple client local models into a global model through collaborative training using weights. The generalized global model obtained from the server is downloaded to each client, and skeletal point action recognition is performed on each client's privacy data (which consists of skeletal point data unique to the client). The server's final training objective is to minimize the global objective function, expressed as follows:

[0010]

[0011] Among them, w * and Φ * Represent the encoder parameters and projection layer parameters of the client-side local model, respectively, |D k | and |D| represent the amount of privacy data of the k-th client and the sum of the amounts of data of the k clients, respectively. kThis represents the training loss of the k-th client-side local model.

[0012] Each training round of the method includes four steps A, B, C, and D. Compared to previous methods, this method mainly focuses on the regularization term Γ in step B and the global model aggregation weight β in step D. k The computational strategy has been improved, and each step is explained in detail below:

[0013] Step A: If it is the first round of training, the parameters of the global encoder are randomly initialized; otherwise, the parameters of the global encoder are aggregated from the parameters of multiple trained client encoders, and the resulting global encoder parameters are used to initialize the parameters of the client local encoders in the next round of training.

[0014] {w1,…,w k ,…,w K}=w g

[0015] Among them, w g This represents the parameters of the global encoder.

[0016] Step B: During local training, the parameters of each client-side local model are updated using the Barlow Twins skeleton point contrast learning framework. The client-side encoder and projection layer are trained based on privacy-preserving data, with the training objective being to minimize the following objective function:

[0017] L(w * oΦ * ) = L BT (w * oΦ * )+Γ

[0018] Where L BT Γ and Γ represent the contrastive learning loss and the regularization term, respectively. The following section details how to obtain the contrastive learning loss L. BT And regularization term Γ:

[0019] (Comparison of learning loss L) BT (Obtained through the following steps)

[0020] Step B01: The original 3D skeleton point sequence x is augmented to obtain x. 1 and x 2 .

[0021] Step B02: x 1 and x 2 They are fed into the encoder separately to extract features h. 1 and h 2 .

[0022] Step B03: Feature h1 and h 2 The feature z is obtained by feeding it into a nonlinear projection layer. 1 and z 2 Used for comparative learning.

[0023] Step B04: Calculate feature z 1 and z 2 The cross-correlation matrix C, and the elements of the matrix C ij Obtained using the following formula:

[0024]

[0025] Where b represents the index of the batch sample, and i and j represent the indexes of the feature dimensions.

[0026] Step B05: The contrastive learning objective function promotes feature z by minimizing the distance between the cross-correlation matrix and the identity matrix. 1 and z 2 The consistency between them is determined by the following comparison learning objective function definition:

[0027]

[0028] Where λ is a positive number, and C is the characteristic z. 1 and z 2 The cross-correlation matrix.

[0029] (The regularization term Γ is obtained through the following steps)

[0030] Step B06: Separate the parameters of the locally trained client encoder and projection layer.

[0031] Step B07: By narrowing the distance between the current client encoder parameters and the global model encoder parameters, while simultaneously widening the distance between the current client encoder parameters and the encoder parameters from the previous round of this client, the generalized encoder regularization Γ is obtained. e :

[0032]

[0033] Where |·|, t, w k w g Let represent K2 regularization, the parameters of the t-th training round, the parameters of the k-th client local encoder, and the parameters of the global encoder, respectively.

[0034] Step B08: Obtain the specific projection layer regularization Γ by narrowing the distance between the current client's projection layer parameters and the previous round's projection layer parameters, while simultaneously widening the distance between the current client's projection layer parameters and the current projection layer parameters of other clients. p :

[0035]

[0036] Where Φ k Φ k′ These represent the projector parameters for clients k and k′, respectively. k′ is randomly selected from other clients.

[0037] Step B09: Joint Encoder-Generalization Regularization (EGR) e Projector-Specialty Regularization (PSR) p The overall regularization term Γ is obtained:

[0038] Γ=λΓ e +ηΓ p

[0039] Where λ and η represent the proportional coefficients used to control EGR and PSR, respectively.

[0040] Step C: Upload the locally trained client encoder to the server for subsequent global model aggregation. The projection layer is still trained only on the client and is not uploaded.

[0041] Step D: The multiple client encoders uploaded to the server are aggregated into a global model by weight summation, as shown in the following formula:

[0042]

[0043] Where S k The aggregate weight matrix S is represented by ⊙, where ⊙ represents matrix dot product. k It is obtained through the divergence of singular value decomposition parameters, and the specific steps are as follows:

[0044] Step D01: Define the parameter matrices of the k-th client local model and the l-th layer of the global model after t-1 rounds of training as follows: and Through calculation and The squared difference between corresponding elements of the matrices is used to measure the parameter divergence between them, resulting in a matrix Λ. l :

[0045]

[0046] Step D02: Next, matrix Λ l Perform singular value decomposition:

[0047] U∑V T =Λl

[0048] U = [U1, ..., U p ]

[0049]

[0050] V = [V1, ..., V p ]

[0051] Among them, matrix unitary matrix U1 is a column vector of matrix U; diagonal matrix For matrix Λ l A non-zero singular value, a unitary matrix V1 is a column vector of matrix V.

[0052] Step D03: Select before The largest singular value and its corresponding singular value vector are taken as the most relevant commonalities between the parameters of the client model and the global model, resulting in the following matrix:

[0053]

[0054]

[0055]

[0056] Step D04: Matrix and Multiply to produce an approximate matrix of the original parametric divergence matrix.

[0057]

[0058] The values ​​in the matrix represent the magnitude of the common correlation between the corresponding parameters.

[0059] Step D05: Matrix The weights are fed into the activation function σ and normalized to obtain the final weight matrix S. k :

[0060]

[0061]

[0062] S k ={s kij}

[0063] Beneficial effects: Compared with the prior art, the improved global model aggregation federated self-supervised skeleton point action recognition method provided by the present invention can greatly enhance the generalization of the local model encoder, and in the process of global model aggregation, it can seek the commonalities between the client local model parameters and the global model parameters, while also retaining certain differences, thereby obtaining a more comprehensive global model.

[0064] The described federated self-supervised skeletal action recognition method achieved optimal performance on multiple public datasets, as shown in Tables 1 and 2. This invention can be applied to many fields such as intelligent monitoring and human-computer interaction, and has broad application prospects. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the overall framework of the method described in this invention;

[0066] Figure 2 This is a schematic diagram of the specific projector regularization and singular value decomposition parameter divergence weighting strategy described in this invention. Detailed Implementation

[0067] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0068] This invention discloses an improved federated self-supervised skeleton point action recognition method for global model aggregation. The method uses a distributed training paradigm to collaboratively train multiple client models and aggregate them into a general global model on the server side.

[0069] Figure 1 The diagram shown illustrates the overall framework of the method. Given K clients, each client possesses its own non-independent and identically distributed privacy data (denoted by D). * (This is an example of a process where) each client-side local model is trained locally using its own private data. After the local client-side models are trained, the encoders of multiple client-side local models are uploaded to the server and aggregated into a global model using weights. The final training objective is to minimize the global objective function, which is expressed as follows:

[0070]

[0071] Among them, w * and Φ * Represent the encoder parameters and projection layer parameters of the client-side local model, respectively, |D k| and |D| represent the amount of privacy data of the k-th client and the sum of the amounts of data of the k clients, respectively. k This represents the training loss of the k-th client-side local model.

[0072] Each training round of the described method includes four steps: A, B, C, and D. Compared to previous methods, this method mainly improves the calculation strategy of the regularization term Γ in step B and the global model aggregation weights in step D, such as... Figure 2 As shown below. Each step is explained in detail below:

[0073] Step A: If it is the first round of training, the parameters of the global encoder are randomly initialized; otherwise, the parameters of the global encoder are aggregated from the parameters of multiple trained client encoders, and the resulting global encoder parameters are used to initialize the parameters of the client local encoders in the next round of training.

[0074] {w1,…,w k ,…,w K}=w g

[0075] Among them, w g This represents the parameters of the global encoder.

[0076] Step B: During local training, the parameters of each client-side local model are updated using the Barlow Twins skeleton point contrast learning framework. The client-side encoder and projection layer are trained based on privacy-preserving data, with the training objective being to minimize the following objective function:

[0077] L(w * oΦ * ) = L BT (w * oΦ * )+Γ

[0078] Where L BT Γ and Γ represent the contrastive learning loss and the regularization term, respectively. The following section details how to obtain the contrastive learning loss L. BT And regularization term Γ:

[0079] (Comparison of learning loss L) BT (Obtained through the following steps)

[0080] Step B01: The original 3D skeleton point sequence x is augmented to obtain x. 1 and x 2 .

[0081] Step B02: x 1 and x 2 They are fed into the encoder separately to extract features h. 1 and h2 .

[0082] Step B03: Feature h 1 and h 2 The feature z is obtained by feeding it into a nonlinear projection layer. 1 and z 2 Used for comparative learning.

[0083] Step B04: Calculate feature z 1 and z 2 The cross-correlation matrix C, and the elements of the matrix C ij Obtained using the following formula:

[0084]

[0085] Where b represents the index of the batch sample, and i and j represent the indexes of the feature dimensions.

[0086] Step B05: Comparatively learn the objective function by minimizing feature z 1 and z 2 The distance between the cross-correlation matrix and the identity matrix is ​​used to promote the feature z. 1 and z 2 The consistency between them is determined by the following comparison learning objective function definition:

[0087]

[0088] Where λ is a positive number, and C is the characteristic z. 1 and z 2 The cross-correlation matrix.

[0089] (The regularization term Γ is obtained through the following steps)

[0090] Step B06: Separate the parameters of the locally trained client encoder and projection layer.

[0091] Step B07: By narrowing the distance between the current client encoder parameters and the global model encoder parameters, while simultaneously widening the distance between the current client encoder parameters and the encoder parameters from the previous round of this client, the generalized encoder regularization Γ is obtained. e :

[0092]

[0093] Where |·|, t, w k w g Let represent L2 regularization, the parameters of the t-th training round, the parameters of the k-th client local encoder, and the parameters of the global encoder, respectively.

[0094] Step B08: Obtain the specific projection layer regularization Γ by narrowing the distance between the current client's projection layer parameters and the previous round's projection layer parameters, while simultaneously widening the distance between the current client's projection layer parameters and the current projection layer parameters of other clients. p :

[0095]

[0096] Where Φ k Φ k ' and k' represent the projector parameters for clients k and k', respectively. k' is randomly selected from other clients.

[0097] Step B09: Joint generalized encoder regularization Γ e and specific projection layer regularization Γ p The overall regularization term Γ is obtained:

[0098] Γ=λΓ e +ηΓ p

[0099] Where λ and η represent the proportional coefficients used to control EGR and PSR, respectively.

[0100] Step C: Upload the locally trained client encoder to the server for subsequent global model aggregation. The projection layer is still trained only on the client and is not uploaded.

[0101] Step D: The multiple client encoders uploaded to the server are aggregated into a global model by weight summation, as shown in the following formula:

[0102]

[0103] Where S k The aggregate weight matrix S is represented by ⊙, where ⊙ represents matrix dot product. k It is obtained through the divergence of singular value decomposition parameters, and the specific steps are as follows:

[0104] Step D01: Define the parameter matrices of the k-th client local model and the l-th layer of the global model after t-1 rounds of training as follows: and Through calculation and The squared difference between corresponding elements of the matrices is used to measure the parameter divergence between them, resulting in a matrix Λ. l :

[0105]

[0106] Step D02: Next, matrix Λ l Perform singular value decomposition:

[0107] U∑V T =Λ l

[0108] in

[0109] U = [U1, ..., U p ]

[0110]

[0111] V = [V1, ..., V p ]

[0112] Step D03: Select before The largest singular value and its corresponding singular value vector are taken as the most relevant commonalities between the parameters of the client model and the global model, resulting in the following matrix:

[0113]

[0114]

[0115]

[0116] Step D04: Matrix and Multiply to produce an approximate matrix of the original parametric divergence matrix.

[0117]

[0118] The values ​​in the matrix represent the magnitude of the common correlation between the corresponding parameters.

[0119] Step D05: Matrix The weights are fed into the activation function σ and normalized to obtain the final weight matrix S. k :

[0120]

[0121]

[0122] S k ={s kij}

[0123] As shown in Tables 1 and 2, the method of this invention achieves optimal performance on multiple datasets. This technology can be applied to many fields such as intelligent monitoring and human-computer interaction. For example, applying the improved global model aggregation federated self-supervised skeleton point action recognition to the field of intelligent monitoring, and achieving client model parameter sharing through distributed collaborative training, allows the aggregation of multiple client model parameters to obtain a more generalized global model. This effectively protects the privacy and security of video data on different clients, thus this technology has high promotional value.

[0124] Among them, FedAvg is the 2017 method "Local learning also matters: Rethinking data heterogeneity in federated learning", FedProx is the 2020 method "Federated optimization in heterogeneous networks", MOON is the 2021 method "Model contrast federated learning", FedBN is the 2021 method "Federated learning based on non-independent identically distributed features with local batch normalization", FedU is the 2021 method "Collaborative unsupervised visual representation learning based on discrete data", Loss is the 2022 method "End-to-end speech recognition based on federated acoustic models", EUC is the 2023 method "Hierarchical adaptive model aggregation for scalable federated learning", and L-DAWA is the 2023 method "Hierarchical divergence-aware weight aggregation in federated self-supervised visual representation learning".

[0125] Table 1

[0126] FedAvg 61.5 41.6 FedProx 62.8 43.5 MOON 60.8 42.9 FedBN 61.1 42.3 Loss 58.3 41.9 L-DAWA 63.2 44.7 FedIGA (Ours) 64.5 50.2

[0127] Table 2

[0128] FedAvg 65.02 35.70 FedU 64.55 35.25 Loss 57.12 34.76 EUC 63.60 35.44 L-DAWA 69.31 41.85 FedIGA (Ours) 70.20 42.91

[0129] This invention provides an approach to improving the implementation of federated self-supervised skeletal point action recognition using global model aggregation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A federated self-supervised skeleton point action recognition method for improving global model aggregation, characterized in that, The method employs a distributed training paradigm. Given K clients, each client possesses its own non-independent, identically distributed privacy data. Each client's local model is trained locally using its own privacy data. The server aggregates multiple client local models into a global model through weighted collaborative training. The generalized global model obtained from the server is downloaded to each client, and skeletal point motion recognition is performed on each client's privacy data. The privacy data consists of skeletal point data unique to each client. The server's final training objective is to minimize the global objective function, expressed as follows: wherein, and denote the encoder parameters and the projection layer parameters of the client local model, respectively, and denote the number of privacy data of the kth client and the sum of the number of K client data, respectively, denotes the training loss of the kth client local model; Multiple client-side encoders uploaded to the server are aggregated into a global model through weight summation, as shown in the following formula: in Represents the aggregate weight matrix. Represents matrix dot product; aggregates weight matrix It is obtained through the divergence of singular value decomposition parameters, and the specific steps are as follows: Step D01: Define the local model and global model of the k-th client after t-1 rounds of training. The parameter matrix of the layer is and Through calculation and The squared difference between corresponding elements of the matrices is used to measure the divergence of their parameters, resulting in a matrix. : Step D02: Next, the matrix Perform singular value decomposition: Among them, matrix unitary matrix , For matrix A column vector; a diagonal matrix , For matrix A non-zero singular value, a unitary matrix , For matrix A column vector; Step D03: Select before The largest singular value and its corresponding singular value vector are taken as the most relevant commonalities between the parameters of the client model and the global model, resulting in the following matrix: Step D04: Matrix Multiply to produce an approximate matrix of the original parametric divergence matrix. : The values ​​in the matrix represent the magnitude of the common correlation between the corresponding parameters; Step D05: Matrix Sent into the activation function The final weight matrix is ​​obtained by normalization. : The formalized expression is as follows: 。 2. The improved global model aggregation-based federated self-supervised skeletal point action recognition method according to claim 1, characterized in that, Each round of training described in this method includes steps A, B, C, and D, as follows: Step A: If it is the first round of training, the parameters of the global encoder are initialized randomly; otherwise, the parameters of the global encoder are aggregated from the parameters of multiple trained client encoders, and the obtained global encoder parameters are used to initialize the parameters of the client local encoder in the next round of training. Step B: During local training, the parameters of each client-side local model are updated using the BarlowTwins skeleton point contrast learning framework; the client-side encoder and projection layer are trained based on privacy data; Step C: Upload the locally trained client encoder to the server for subsequent global model aggregation. The projection layer is still trained only on the client and not uploaded. Step D: The multiple client encoders uploaded to the server are aggregated into a global model by weight summation.

3. The improved global model aggregation-based federated self-supervised skeletal point action recognition method according to claim 2, characterized in that, In step B, the training objective is to minimize the following objective function: in and represents the contrastive learning loss and the regularization term, respectively.

4. The improved global model aggregation-based federated self-supervised skeletal point action recognition method according to claim 3, characterized in that, Comparative learning loss Obtain it through the following steps: Step B01: Convert the original 3D bone point sequence Data augmentation and ; Step B02: and They are fed into the encoder separately to extract features. and ; Step B03: Features and Features are obtained by feeding them into a nonlinear projection layer. and Used for comparative learning; Step B04: Calculate features and The cross-correlation matrix C, the elements of the matrix Obtained using the following formula: Where b represents the index of the batch sample, and i and j represent the indexes of the feature dimensions; Step B05: The contrastive learning objective function promotes feature recognition by minimizing the distance between the cross-correlation matrix and the identity matrix. and The consistency between them is determined by the following comparison learning objective function definition: Where λ is a positive number, and C is the characteristic. and The cross-correlation matrix.

5. The improved global model aggregation-based federated self-supervised skeletal point action recognition method according to claim 3, characterized in that, Regularization term Obtain it through the following steps: Step B06: Separate the parameters of the locally trained client encoder and projection layer; Step B07: Generalized encoder regularization is achieved by narrowing the distance between the current client encoder parameters and the global model encoder parameters, while simultaneously widening the distance between the current client encoder parameters and the encoder parameters from the previous round of this client. : in ,t, , They represent Regularization, parameters of the t-th training round, parameters of the k-th client local encoder, parameters of the global encoder; Step B08: Obtain specific projection layer regularization by narrowing the distance between the current client's projection layer parameters and the previous projection layer parameters of the same client, while simultaneously widening the distance between the current client's projection layer parameters and the current projection layer parameters of other clients. : in , They represent the client respectively. and Projector parameters; It was randomly selected from other clients; Step B09: Joint generalized encoder regularization and specific projection layer regularization Obtain the overall regularization term. : in, These represent the proportional coefficients used to control EGR and PSR, respectively.

Images