Non-learnable sample generation method and privacy protection method for bone action privacy protection

Abstract

The application discloses a non-learnable sample generation method and a privacy protection method for bone action privacy protection, and relates to the fields of computer vision and privacy calculation. Bone data is acquired; noise initialization is performed according to the bone data; topological relation extraction is performed on the bone data; the generated noise data and the extracted topological relation are aligned to obtain aligned noise data; constraint optimization is performed on the aligned noise data; the noise data after the constraint optimization is added to the bone data to generate a non-learnable sample; and a bone action recognition model is trained by using the non-learnable sample, so that the privacy of the bone data is protected. The noise generation algorithm designed in the application makes the topological structure of the noise data consistent with that of the original bone data, solves the problem of unstable performance of a traditional method on bone data, and provides effective support for preventing a third party from misusing bone data for model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and privacy computing, in particular to a non-learnable sample generation method and a privacy protection method for skeletal action privacy protection. BACKGROUND

[0002] In recent years, the rapid development of skeletal action recognition technology, including the breakthroughs of STGCN[1], CTRGCN[3] and STTFormer[6] models, has significantly improved the accuracy of personal behavior analysis based on skeletal data. In particular, the progress of multi-modal spatio-temporal modeling technology has exacerbated public concerns about the leakage of motion data privacy. High-precision skeletal action recognition systems can have a dual social impact: in the field of medical rehabilitation, it can assist in the diagnosis of movement disorders such as Parkinson's disease; but in the security monitoring scene, unauthorized behavior analysis may infringe on personal privacy. For example, through skeletal data, certain professional habits (such as the operating posture of medical staff) can be inferred, or in the financial scene, user identity features (such as gait recognition) can be leaked. More seriously, commercial human action datasets can be used to train deep monitoring systems, leading to the malicious use of personal behavior characteristics.

[0003] To address these risks, researchers have proposed non-learnable sample technology (such as EM[2]), which adds specific noise to the training data to prevent skeletal action recognition models from learning effective features. Current protection schemes mainly fall into two categories: passive defense based on anonymization (such as fuzzing[5]) and active defense based on data poisoning (such as EM[2]). However, the time-series characteristics and low degree of freedom of skeletal data make it difficult to directly apply traditional image protection methods.

[0004] The existing skeletal data protection methods mainly have the following three limitations:

[0005] (1) Topology Ignoring Problem: Existing non-learnable sample generation methods (such as EM[2], LSP[4]) are mainly designed for images and do not consider the unique joint topology relationship of skeletal data. The noise generated by traditional methods destroys the natural connectivity between human joints, reducing the protection effect and increasing the visual abnormality;

[0006] (2) Efficiency Bottleneck: Model-dependent methods (such as TAP[3]) require iterative optimization of noise, which requires more than 250,000 seconds of computation time for large-scale datasets such as NTU120, while model-independent methods (such as LSP[4]) have improved efficiency but reduced protection effectiveness by about 38%;

[0007] (3) Defense Vulnerability: Existing methods lack robustness to data augmentation (such as rotation, Gaussian noise) and adversarial training. For example, the EM[2] method's accuracy rebounds to 37.39% under adversarial training, severely undermining the protection effect.

[0008] In addition, it is also particularly noteworthy that the current evaluation system has the limitation of "static test", and most studies only verify the performance on standard test sets without considering the dynamic attack scenarios that may be encountered in actual applications (such as cross-model transfer attacks [3]). This leads to the fact that the protection effect reported in the literature is higher than the actual application performance.

[0009] The relevant references are as follows:

[0010] [1] S. Yan, Y. Xiong, and D. Lin, "Spatial temporal graphconvolutional networks for skeleton-based action recognition," in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.

[0011] [2] H. Huang, Y. Wang, Z. Chen, and Y. Tang, "Unlearnable examples,"in Proceedings of the International Conference on Learning Representations,2021.

[0012] [3] L. Fowl, M. Goldblum, P. Chiang, J. Geiping, W. Czaja, and T.Goldstein, "Adversarial poisoning," in Advances in Neural InformationProcessing Systems, vol. 34, 2021.

[0013] [4] D. Yu, A. Zhang, K. Liu, J. Li, and Q. Xu, "LSP," in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and DataMining, 2022.

[0014] [5] S. Moon, Y. Park, D. Kim, and J. Choi, "Skeleton anonymization," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023.

[0015] [6] H. Qiu, Y. Hou, and M. Li, "STTFormer," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022.

[0016] [7] Chen S, Yuan G, Cheng X, et al. Self-Ensemble Protection:Training Checkpoints Are Good Data Protectors[C] / / The Eleventh International Conference on Learning Representations. SUMMARY

[0017] In order to overcome the defects in the prior art described above, the present application provides a non-learnable sample generation method for skeleton action privacy protection, a privacy protection method,

[0018] In order to achieve the above purpose, the present application adopts the following technical scheme, comprising:

[0019] The non-learnable sample generation method for skeleton action privacy protection comprises the following steps:

[0020] S1, obtaining skeleton data;

[0021] S2, the left and right branches are respectively processed: the left branch is initialized with noise according to the skeleton data; the right branch extracts the topological relationship of the skeleton data;

[0022] S3, aligning the noise data generated by the left branch and the topological relationship extracted by the right branch to obtain aligned noise data;

[0023] S4, performing constraint optimization on the aligned noise data of step S3;

[0024] S5, adding the noise data after constraint optimization of step S4 to the skeleton data to generate a non-learnable sample.

[0025] Preferably, in step S1, each bone action category corresponds to a bone data set, and each bone data set contains a plurality of bone data; the bone data is composed of a bone action sequence, and the format is ; wherein,

[0026] C is the number of coordinate channels of the joint, including three channels for describing the position information of the joint;

[0027] T is the number of time frames, i.e. the time length of the bone action sequence;

[0028] V is the number of joints;

[0029] M is the number of bone instances, i.e. the number of characters.

[0030] Preferably, in step S2, the left branch is initialized with noise according to the bone data, and the specific process is as follows:

[0031] The time sequence dimension and the joint dimension of the noise data are calculated, and the calculation method is as follows:

[0032] ;

[0033] ;

[0034] wherein, represents the floor function, , are the time patch size and the joint patch size, respectively;

[0035] A noise set is generated by normal distribution, one noise set is generated for each bone action category, n is the number of samples in the noise set D , i.e. the number of noises;

[0036] Random numbers are uniformly sampled from [-1, 1] to construct a random matrix A , which is used to add covariance in the noise set D to obtain a noise set with added covariance, and the adding method is as follows:

[0037] ;

[0038] By adding the cluster center of the noise set to each row of the noise set c , the noise set Mapping to the vertices of the hypercube, ensuring that each noise is within a reasonable range and that there is linear separability between different noise sets, obtaining the mapped noise set , the mapping method is:

[0039] .

[0040] Preferably, in step S2, the right branch extracts the topological relationship of the bone data, as follows:

[0041] For bone data, the cosine similarity between joints is calculated to quantify the topological correlation of bone movement, and the calculation method is:

[0042] ;

[0043] wherein, and represent the index of the joint; is the number of coordinate channels of the joint, including three channels; and represent the values of the joints and in the first coordinate channel; represent the cosine similarity between joints and , i.e. the correlation value;

[0044] According to the cosine similarity between joints, a correlation matrix is constructed.

[0045] Preferably, step S3 is as follows:

[0046] S31, copy the noise M times, adjust the dimension of the noise data to , align the number of bone instances of the bone data;

[0047] S32, copy each time sequence of the noise times, adjust the dimension of the noise data to , give the same noise value to the continuous frames of noise, and align the number of time frames of the bone data;

[0048] S33, according to the correlation matrix, group the joints according to the correlation, into groups of associated joints, inject noise into the associated joint groups, adjust the dimension of the noise data to , and give the same noise value to the joints in each associated joint group, aligning the number of joints of the bone data.

[0049] Preferably, step S4 is as follows:

[0050] limiting the noise amplitude with an L∞ norm;

[0051] normalizing the noise data.

[0052] The application further provides a skeleton action privacy protection method, which utilizes the non-learnable sample generation method for skeleton action privacy protection to generate non-learnable samples, and utilizes the non-learnable samples to train a skeleton action recognition model, so as to realize privacy protection of skeleton data.

[0053] The application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions realize the non-learnable sample generation method for skeleton action privacy protection when executed by a processor.

[0054] The application further provides a readable storage medium, which stores a computer program, and the computer program realizes the skeleton action privacy protection method when executed.

[0055] The application further provides an electronic device, which comprises a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor realizes the skeleton action privacy protection method when executing the computer program.

[0056] The application has the following advantages:

[0057] (1) The application first discovers that the performance of non-learnable noise of skeleton data is closely related to the topological relationship thereof, and the performance is positively correlated with the topological consistency, thereby creating a topological consistency noise generation mechanism, generating noise conforming to the human body structure by utilizing the cosine correlation between skeleton joints, and designing a noise generation algorithm to make the topological structure of the noise consistent with the original skeleton data, thereby solving the problem of unstable performance of traditional methods on skeleton data.

[0058] (2) The application proposes a first model-independent method for skeleton data, and improves the effect by maintaining the consistency of noise and skeleton topological structure.

[0059] (3) The application develops an efficient noise generation algorithm, and the time cost is much lower than that of a model-dependent method.

[0060] (4) The application designs a three-stage processing flow, i.e., generating an initial disturbance cluster, calculating the average topological relationship of the skeleton, and introducing local correlation based on the topological relationship.

[0061] (5) The application has strong robustness for data enhancement and adversarial training.

[0062] (6) It has been verified that the method of the present invention achieves the best protection effect on the NTU60 / NTU120 dataset, with an average test accuracy of 4.17-6.37%.

[0063] (7) The method of the present invention provides an effective solution to prevent third parties from abusing skeletal data for model training. It is particularly suitable for scenarios that require protection of personal action privacy (such as medical rehabilitation, security monitoring, etc.), while maintaining the normal use value of the original data, which is of practical significance. Attached Figure Description

[0064] Figure 1 This is a flowchart of the non-learnable sample generation method for protecting skeletal motion privacy according to the present invention.

[0065] Figure 2 This is a schematic diagram illustrating the generation of unlearnable samples in this invention.

[0066] Figure 3 This is a heatmap comparison of the cosine similarity between the joint points of the method of the present invention and existing methods. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] In this invention, the related technologies have the following meanings:

[0069] (1) Unlearnable samples: By adding perturbations (||δ||) that are imperceptible to the human eye but can disrupt model training. ∞ ≤ ), where ||·|| ∞ The infinite norm constraint is represented by δ, which is the perturbation. The perturbation threshold is used to make the model's performance on the clean test set Dt significantly decrease after training on the poisoned data Du.

[0070] (2) Topological consistency: This means that the noise disturbance should maintain the same joint association pattern as the original skeletal data, that is, the thermal of the effective noise should have a similar spatial correlation distribution as the original data;

[0071] (3) Model-independent method: It does not rely on the proxy model to generate perturbations. It directly synthesizes noise by analyzing the inherent characteristics of the data (such as the linear separability of LSP and the topological relationship of TUE). It has the advantage of O(n) time complexity, where O(n) means that the execution time increases linearly with the input size.

[0072] (4) Skeleton data characteristics: taking the NTU-RGB+D format as an example, it contains the three-dimensional coordinates of 25 joints, and has a four-dimensional structure (coordinate channel, time, joint, and person);

[0073] (5) Model-dependent method: refers to the specific technical framework, algorithm logic, data processing flow or theoretical hypothesis relied on when constructing, training, optimizing or evaluating a model, which determines how the model learns the rules from the data, how it generalizes to new scenarios, and the performance boundary of the model.

[0074] Embodiment 1

[0075] In view of the problems that the existing unlearnable sample technology mainly targets image data, and the unique characteristics (such as low degree of freedom and time sequence dependence) of skeleton data as time sequence data, direct application of image methods does not work well, and the noise generated by the current method does not match the topological structure of the skeleton data, resulting in unstable protection effect, and the noise topological consistency of the existing method on different data sets is significantly different. The present application provides an unlearnable sample generation method for skeleton action privacy protection, as shown in Figure 1 and Figure 2 , comprising the following steps:

[0076] S1, obtaining skeleton data.

[0077] Each skeleton action category corresponds to a skeleton data set, and each skeleton data set contains a plurality of skeleton data. The skeleton data is composed of a skeleton action sequence, and the format is ; wherein,

[0078] C is the number of coordinate channels of the joint, including three channels for describing the position information of the joint;

[0079] T is the number of time frames, i.e. the time length of the skeleton action sequence, each frame corresponds to a timestamp skeleton state, such as a hand waving action of about 150 frames;

[0080] V is the number of joints (i.e. the key nodes constituting the human skeleton), and the NTU standard defines 25 joints;

[0081] M is the number of skeleton instances (i.e. the number of people), i.e. the number of target persons contained in a single skeleton sequence, and the NTU mainly supports single-person / two-person action.

[0082] S2, left and right branches are processed in parallel.

[0083] (1) The left branch initializes noise according to the skeleton data.

[0084] Computing the time dimension of noise data and joint dimension , the calculation method is:

[0085] ;

[0086] ;

[0087] wherein, represents the floor function, , respectively, the time patch size and the joint patch size.

[0088] Generate a noise set by normal distribution , each bone action category corresponds to a noise set, n is the number of samples in the noise set D , that is, the number of noises.

[0089] From uniformly sample random numbers to construct a random matrix A , which is used to add covariance in the noise set D , to obtain the noise set after adding covariance , the adding method is:

[0090] ;

[0091] By adding the cluster center of the noise set to each row of the noise set c , the noise set is mapped to the vertices of the hypercube, ensuring that each noise is within a reasonable range (distributed near the vertices of the hypercube) and that different noise sets have linear separability (simple linear models can be used to classify noises), to obtain the mapped noise set , the mapping method is:

[0092] .

[0093] (2) The right branch extracts the topological relationship of the skeleton data.

[0094] For skeleton data, the cosine similarity between joints is calculated to quantify the topological correlation of skeletal motion. The cosine similarity calculation process can be represented as:

[0095] ;

[0096] wherein, and represent the index of the joint; is the number of coordinate channels of the joint, including Three channels; and Indicates key points and In the The values ​​of each coordinate channel; Indicates key points and The cosine similarity between the two is the correlation value. The cosine similarity value ranges from [-1, 1]. The larger the value, the greater the correlation.

[0097] A correlation matrix is ​​constructed based on the cosine similarity between joints, a heatmap is generated, and strongly correlated regions (such as hand-wrist, knee-ankle) are identified.

[0098] S3 aligns the noise data generated by the left branch with the topological relationship extracted by the right branch to obtain aligned noise data.

[0099] Step S3 is as follows:

[0100] S31, copy each noise M times, and adjust the dimension of the noise as follows. The number of bone instances aligned to the skeletal data (action sequence);

[0101] S32, copies each time sequence of each noise. Next, adjust the noise dimension as follows: This makes the noise continuous. Frames are assigned the same noise value to match the temporal continuity of the action and align the number of time frames of the skeletal data.

[0102] S33, Based on the correlation matrix, the key points are grouped according to their correlation. For each associated joint group, inject noise into the associated joint group (e.g., synchronous perturbation of all joints in the left arm), and adjust the noise dimension to... Each joint in a group of related joints is assigned the same noise value to align the number of joints in the skeletal data.

[0103] Among them, stronger consistency constraints are applied to strongly correlated regions in the heatmap (such as spinal joints).

[0104] S4, perform constraint optimization on the noise data after alignment in step S3.

[0105] Step S4 is as follows:

[0106] S41, use the L∞ norm to limit the amplitude of the noise data (ε=0.01) to ensure that the disturbance is not easily detected by the naked eye;

[0107] S42 normalizes the noise data so that its distribution matches the natural range of motion of the skeletal data.

[0108] S5, add the noise data optimized in step S4 to the bone data to generate a topologically consistent non-learnable sample, which retains the structural characteristics of the bone data and can also destroy the model training effect.

[0109] Embodiment 2

[0110] The application also provides a skeleton action privacy protection method, which uses the non-learnable sample generation method for skeleton action privacy protection provided in Embodiment 1 to generate a non-learnable sample, and uses the non-learnable sample to train a skeleton action recognition model, so as to avoid the skeleton action recognition model from learning useful features in the skeleton data, thereby realizing privacy protection of the skeleton data.

[0111] The method of the application provides an effective solution to prevent third parties from misusing skeleton data for model training, and is particularly suitable for scenarios that need to protect personal action privacy (such as medical rehabilitation, security monitoring, etc.), while maintaining the normal use value of the original data, and has practical significance.

[0112] Existing non-learnable sample (UE) technology is mainly aimed at image data, and the application first proposes a topologically aware noise generation method (TUE) for skeleton data. As shown in Figure 3 , Figure 3 The heat map of the correlation between the joints of the method (Ours) of the application and the existing methods (EM[2], TAP[3], LSP[4]) is compared, and by analyzing the topological relationship of the skeleton joints, the noise generation method designed by the application makes the topological structure of the noise data consistent with the original skeleton data, solving the problem of unstable performance of traditional methods on skeleton data.

[0113] The existing model-dependent method needs to iteratively optimize the noise, and the time consumption is as long as tens of thousands of seconds, as shown in Table 1, Table 1 is a comparison table of time cost of the method of the application and the existing methods (EM[2], SEP[7], TAP[3], LSP[4]) on different data sets (NTU60_S, NTU120_S, NTU60_V, NTU120_V), wherein 1k=1000, the method (Ours) of the application shortens the generation time to within 180 seconds by directly synthesizing topologically consistent noise, and the efficiency is improved by more than 200 times, and does not need to rely on proxy model training.

[0114] Table 1

[0115] ;

[0116] As shown in Table 2, Table 2 is a comparison table of test accuracy of the method of the present application (Ours) and existing methods (EM[2], SEP[7], TAP[3], LSP[4]) on different model architectures (STGCN, CTRGCN, MSG3D, 2s-AGCN, STTFormer, FRHEAD) and different data sets (NTU60_S, NTU120_S, NTU60_V, NTU120_V), the average test accuracy of the method of the present application (Ours) on the NTU60_S and NTU120_S data sets is reduced to 6.37% and 4.17% respectively, which is significantly lower than the baseline method (such as 8.48% and 14.34% of EM), and the method of the present application is optimal on 6 different model architectures such as ST-GCN and Transformer.

[0117] Table 2

[0118] ;

[0119] As shown in Table 3, Table 3 is a comparison table of test accuracy of the method of the present application (Ours) and existing methods (EM[2], SEP[7], TAP[3], LSP[4]) on different model architectures (STGCN, CTRGCN, MSG3D, 2s-AGCN, STTFormer, FRHEAD) and different enhancements (rotation Rotate, Gaussian noise GausNoise, GausFilter, Shear), the method of the present application (Ours) still maintains the lowest accuracy under rotation (Rotate) and Gaussian noise (GausNoise) enhancement.

[0120] Table 3

[0121] ;

[0122] As shown in Table 4, Table 4 is a comparison table of accuracy before and after adversarial training of the method of the present application (Ours) and existing methods (EM[2], SEP[7], TAP[3], LSP[4]), the accuracy of the method of the present application (Ours) after adversarial training is only increased from 9.37% to 10.36%, while EM is increased from 10.81% to 37.39%, which proves the strong robustness of the method of the present application (Ours) to defense strategies.

[0123] Table 4

[0124] ;

[0125] The above is only a preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A non-learnable sample generation method for skeletal action privacy protection, characterized in that, The method comprises the following steps: S1, obtaining skeleton data; S2, left and right branches are respectively processed: the left branch performs noise initialization according to the skeleton data; the right branch extracts topological relations of the skeleton data; S3, aligning the noise data generated by the left branch and the topological relations extracted by the right branch to obtain aligned noise data; S4, performing constraint optimization on the noise data aligned in step S3; S5, adding the noise data optimized in step S4 to the skeleton data to generate a non-learnable sample; In step S1, each skeleton action category corresponds to a skeleton dataset, and each skeleton dataset contains a plurality of skeleton data. The skeleton data is composed of a skeleton action sequence, and the format is ; wherein, C is the coordinate channel number of the joint node, including three channels for describing the position information of the joint node; T is the number of time frames, that is, the time length of the skeleton action sequence; V is the number of joints; M is the number of skeleton instances, that is, the number of characters; In step S2, the left branch performs noise initialization according to the skeleton data, which is specifically as follows: Computing the temporal dimension of the noise data and the joint dimension in the following way: ； ； wherein denotes a floor function, p t , p v are a temporal patch size and a joint patch size, respectively; Generating a noise set by normal distribution corresponding to each bone action category, n The number of samples in the noise set D is the number of noises. Random matrix is constructed from uniformly sampling random numbers in [-1, 1] A , for adding covariance in noise set D , to obtain noise set after adding covariance , and the adding mode is ； Through noise set Add noise set to each line Cluster center c , collect noise Mapping to the vertices of the hypercube ensures that each noise is within a reasonable range and that different noise sets are linearly separable, resulting in the mapped noise set. The mapping method is as follows: ； In step S2, the right branch extracts topological relations of the skeleton data, which is specifically as follows: For the skeleton data, the cosine similarity between joints is calculated to quantify the topological correlation of the skeleton motion, and the calculation method is as follows: ； wherein, and denotes the index of the joint; is the number of coordinate channels of the joint, including three channels; and denotes the joint and the value in the first coordinate channel; denotes the cosine similarity, i.e. the correlation value, between the joints and ; A correlation matrix is constructed according to the cosine similarity between joints; Step S3 is specifically as follows: S31, copying the noise M times, adjusting the dimension of the noise data to aligning the number of bone instances of the bone data; S32, copying each timing of the noise Next, adjusting the dimension of the noise data to Making the noise continuous Assigning the same noise value to the frames, aligning the number of time frames of the skeletal data; S33, according to the correlation matrix, grouping the nodes according to the correlation, into a group of associated joints, injecting noise into the group of associated joints, adjusting the dimension of the noise data to the same noise value is given to the nodes in each group of associated joints, and the number of nodes of the skeletal data is aligned.

2. The unlearnable sample generation method for skeleton action privacy protection according to claim 1, characterized in that, Step S4 is specifically as follows: The noise amplitude is limited by using the L∞ norm; The noise data is normalized.

3. A method for privacy protection of skeletal action, characterized by, The non-learnable sample generation method for skeleton action privacy protection according to any one of claims 1-2 is used to generate a non-learnable sample, and the non-learnable sample is used to train a skeleton action recognition model, thereby realizing privacy protection of skeleton data.

4. A computer program product, characterized in that, It includes computer programs / instructions that are executed by a processor to implement the non-learnable sample generation method for skeleton action privacy protection according to any one of claims 1-2.

5. A readable storage medium characterized by, It has a computer program stored thereon, and the computer program is executed to implement the skeleton action privacy protection method according to claim 3.

6. An electronic device, comprising: It includes a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the skeleton action privacy protection method according to claim 3.

Images