A self-supervised behavior recognition method and system based on adversarial training

Abstract

The present invention discloses a self-supervised behavior recognition method and system based on adversarial training. The self-supervised behavior recognition method based on adversarial training includes: extracting video data and inputting it into a standardized flow model; performing attack processing on the video data to obtain an adversarial video; calculating the adversarial loss of the video data and the adversarial video in a learning mode; updating the expected value parameters of the standardized flow model based on the adversarial loss, and judging whether the update is completed; if so, generating adversarial samples based on the standardized flow model and the expected value parameters; and inputting the adversarial samples into comparative learning. The present invention attacks samples based on the adversarial attack of the standardized flow, and also treats the attacked samples as positive samples and adds them to the training of comparative learning for adversarial training, thereby improving the accuracy and robustness of the model.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a self-supervised behavior recognition method and system based on adversarial training. Background Art

[0002] We've noticed that current models for action recognition tasks often rely too heavily on spatial information. We believe action recognition should focus on the action itself (e.g., swimming), rather than identifying an action through context (e.g., a swimming pool). However, in existing datasets, many actions occur in specific contexts, creating a natural bias in the dataset. Training models with such biased datasets can lead to poor performance on certain difficult examples.

[0003] In addition, in the contrastive learning mode, the data augmentation method will affect the pre-training effect of the model, so it is very meaningful to explore more effective data augmentation methods. In summary, a self-supervised behavior recognition method and system based on adversarial training is proposed. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology. The present invention provides a self-supervised behavior recognition method and system based on adversarial training. The adversarial attack based on the standardized flow attacks the samples, and the attacked samples are also treated as positive samples and added to the contrastive learning training for adversarial training, thereby improving the accuracy and robustness of the model.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a self-supervised behavior recognition method based on adversarial training, which includes:

[0006] Extract video data and input it into the standardized flow model;

[0007] Perform attack processing on video data to obtain adversarial video;

[0008] Calculate the adversarial loss of video data and adversarial video in learning mode;

[0009] Update the expected value parameters of the normalized flow model based on the adversarial loss and determine whether the update is complete;

[0010] If so, generate adversarial samples based on the normalized flow model and expected value parameters;

[0011] Feed adversarial examples into contrastive learning.

[0012] Preferably, the extracting video data includes:

[0013] Extract a batch of video data, denoted as x i , input the video data into the standardized flow model and set the number of times to update the mathematical expectation parameter.

[0014] Preferably, the attack processing on the video data includes:

[0015] A frame is randomly extracted from the video data, pixels are generated through the standardized flow model, and added to each frame of the entire video to obtain an adversarial video.

[0016] Preferably, the calculating the adversarial loss of the video data and the adversarial sample in the learning mode includes:

[0017] The formula for calculating the adversarial loss is:

[0018] L(x')=max(0,logC(x') y -max logC(x') c ),c≠y

[0019] Where x' represents the input sample received from the adversarial attack, C() represents the classifier, and logC(x') y represents the output score of the classifier for the yth item, i.e., the correct item, maxlogC(x') c Indicates the maximum score among all classes.

[0020] Preferably, updating the expected value parameters of the normalized flow model based on the adversarial loss includes:

[0021] Use natural selection strategy to repeatedly update expected value parameters;

[0022] According to the lazy man's theorem, the loss function formula of the natural selection strategy in this scenario is expressed as:

[0023]

[0024] Where J(μ,σ) represents the loss in the natural selection strategy, projS represents the corresponding space where the generated adversarial samples are projected in order to meet the requirement of invisibility of adversarial attacks; the right side of this formula represents the loss when the z vector obeys z~N(z|μ,σ 2 I) Under the distribution condition, after k z vectors are input into the normalized flow model f(), and the adversarial samples generated by adding the projs restriction, the mathematical expectation of the loss L of k obtained in the classifier;

[0025] Using the maximum likelihood estimation method for the above formula, the Jacobian determinant of J(μ,σ) can be obtained:

[0026]

[0027] Where, is the Jacobian determinant of J(μ,σ), I is the identity matrix, μ is the expected value parameter, and σ is the hyperparameter;

[0028] Next, update the expected value parameter μ by the following formula:

[0029]

[0030] Where α is an empirical coefficient, I is the identity matrix, μ is the expected value parameter, and σ is the hyperparameter.

[0031] Preferably, the determining whether the update is completed includes:

[0032] Determine whether the expected value parameter update is completed based on the adversarial loss and the preset number of updated mathematical expectation value parameters;

[0033] If the adversarial loss L(x') = 0, the mathematical expectation parameter μ is determined to have completed the update and proceed to the next step;

[0034] If the adversarial loss L(x') ≠ 0 and the number of times the preset mathematical expectation parameter is not reached, it is determined that the update of the mathematical expectation parameter μ is not completed, and the video data is attacked again;

[0035] If the adversarial loss L(x') ≠ 0 and the number of mathematical expectation parameters is reached, it is determined that the mathematical expectation parameter μ has been updated and the next step is performed.

[0036] Preferably, the generating of adversarial samples based on the standardized flow model and the expected value parameter includes:

[0037] In the process of generating adversarial samples, the normalized flow model is used to generate adversarial samples x' from random vector z, where the random vector z obeys z~N(z|μ,σ 2 I), the adversarial sample formula is:

[0038] x'=(projs(f(z)),z~N(z|μ,σ 2 I).

[0039] Where I is the identity matrix, μ is the expected value parameter, and σ is the hyperparameter.

[0040] Preferably, inputting the adversarial sample into contrastive learning includes:

[0041] In the standard contrastive learning model, a third branch is added, which consists of constructed adversarial samples, and the adversarial samples are used as positive samples for data enhancement.

[0042] Preferably, the cosine distance is used to represent the similarity between samples in the contrastive learning, and the attack is deemed to be successful when the cosine distance between positive samples is less than the minimum value of the cosine distances between positive and negative samples.

[0043] Accordingly, the present invention further provides a self-supervised behavior recognition system based on adversarial training, wherein the self-supervised behavior recognition system is used in the above-mentioned self-supervised behavior recognition method based on adversarial training, and the self-supervised behavior recognition system comprises:

[0044] Extraction module: extracts video data and inputs it into the standardized flow model;

[0045] Attack module: performs attack processing on video data to obtain adversarial video;

[0046] Technical module: Calculate the adversarial loss of video data and adversarial videos in learning mode;

[0047] Update module: updates the expected value parameters of the normalized flow model based on the adversarial loss and determines whether the update is completed;

[0048] Adversarial generation module: Generates adversarial samples based on the normalized flow model and expected value parameters;

[0049] Attack judgment module: input adversarial samples into contrastive learning.

[0050] Embodiments of the present invention provide a self-supervised behavior recognition method and system based on adversarial training. This method attacks samples using an adversarial attack based on a normalized flow. These samples are then treated as positive samples and incorporated into contrastive learning training, thereby increasing the number of positive samples. Furthermore, because adversarial samples only alter a few pixels and leave semantic information unchanged, they increase the distance between positive samples. Therefore, adversarial attacks can be used as an efficient data augmentation method and a method for constructing difficult examples. Furthermore, because the images generated by the normalized flow have a distribution similar to that of the original image, the image generated for a specific frame of a video in this method can be considered the video background. This background addition makes the image more likely to be used as a secondary sample, thereby reducing the impact of background bias in the dataset during training on model performance. Furthermore, training using adversarial attacks can improve model robustness. Training and testing using the behavior recognition datasets UCF-101 and HMDB-51 show that adversarially trained network models achieve higher recognition rates than non-adversarially trained network models. Furthermore, when subjected to the corresponding adversarial attacks, the adversarially trained network exhibits a lower misclassification probability, demonstrating the enhanced robustness of the adversarially trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 4 is a flow chart of a self-supervised behavior recognition method in an embodiment of the present invention.

[0053] Figure 2 Schematic diagram of training a standardized flow model in an embodiment of the present invention.

[0054] Figure 3 2 is a schematic diagram of the structure of adversarial training in an embodiment of the present invention.

[0055] Figure 4 2 is a schematic diagram of the structure of the self-supervised behavior recognition system in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] Example

[0058] Figure 1 FIG. 4 shows a flow chart of a self-supervised behavior recognition method according to an embodiment of the present invention. Figure 2 FIG2 shows a schematic diagram of training a standardized flow model in an embodiment of the present invention. Figure 3 The schematic diagram of the structure of adversarial training in an embodiment of the present invention is shown. The self-supervised behavior recognition method based on adversarial training includes:

[0059] S1: Extract video data and input it into the standardized flow model.

[0060] To conduct adversarial training in contrastive learning mode, we need to construct adversarial samples in contrastive learning. Here we use AdvFlow as the adversarial attack method. Extract a batch of video data, denoted as x i , input the video data into the standardized flow model and set the number of times to update the mathematical expectation parameter.

[0061] Pre-trained standardized flow generation model f(). The standardized flow model is a generation model that can generate outputs that conform to a certain probability distribution from simple distributions, such as random vectors. A data set D contains many data samples x, and these data samples x obey the probability density function f. θ (x), the data sample x is randomly sampled from this distribution. The generative model is generated by θ (x) model, and use this model to generate x that obeys the fθ(x) distribution. But the probability density distribution of the real data set is f θ (x) is very complex and does not simply obey the normal distribution or other common distributions, so f θ (x) is difficult to model. Normalizing flows methods use multiple transformations (x = z k =f k *f k-1 *...*f1(z0)) gradually complicates a simple distribution (such as the standard normal distribution) to obtain the desired distribution. Pre-train a reversible normalized flow model that can generate images close to the target image distribution from a random distribution.

[0062] S2: Perform attack processing on the video data to obtain an adversarial video.

[0063] In the video data x i Each video in the random extraction of a frame, through the normalized flow model to generate some pixels x dist , will add pixel x dist To each frame of the entire video, the entire batch after the attack is called x adv That is, adversarial video.

[0064] S3: Calculate the adversarial loss of video data and adversarial samples in learning mode.

[0065] C&W loss was originally proposed in the context of supervised learning. Assuming the model is a classifier that scores different labels, for model x, its correct label is y. For adversarial attacks without a specified target, C&W loss can be expressed as:

[0066] L(x')=max(0,logC(x') y -max logC(x') c ),c≠y

[0067] Where x' represents the input sample received from the adversarial attack, C() represents the classifier, and logC(x') y represents the output score of the classifier for the yth item, i.e., the correct item, maxlogC(x') cIndicates the maximum score among all classes.

[0068] Calculate the adversarial loss C&Wloss of the vectors output after the samples under adversarial attack and the clean samples are input into the network. In the adversarial attack, it is necessary to optimize the adversarial loss C&W loss, so it is necessary to search for the gradient of the adversarial loss C&W loss. Since the adversarial attack in this article takes the form of a black-box attack, the black-box attack requires that the model structure is unknown to the adversarial attack algorithm, and only the input and output of the model are known, so the layer-by-layer gradient descent method cannot be used. Here, the natural selection strategy is used to update the gradient. First, a perturbation is added to the mathematical expectation parameter μ to generate k perturbed random vectors z. The k random vectors z are used to generate k adversarial samples x' through the normalized flow model, and are superimposed on the original image to form k attacked samples x adv , put k x adv Get k classification score results z through the encoder adv .

[0069] S4: Update the expected value parameters of the normalized flow model based on the adversarial loss and determine whether the update is complete.

[0070] Use natural selection strategy to repeatedly update expected value parameters;

[0071] According to the rule of the lazy statistician, the loss function formula of the natural selection strategy in this scenario can be expressed as:

[0072]

[0073] Where J(μ,σ) represents the loss in the natural selection strategy, projS represents the corresponding space where the generated adversarial samples are projected in order to meet the requirement of invisibility of adversarial attacks; the right side of this formula represents the loss when the z vector obeys z~N(z|μ,σ 2 I) Under the distribution condition, after k z vectors are input into the normalized flow model f(.), and the adversarial samples generated by adding the projs restriction, the mathematical expectation of the loss L of k obtained in the classifier;

[0074] In order to find the gradient value, we need to find the Jacobian determinant J(μ,σ). But It is impossible to directly calculate the partial derivative. So we need to use the maximum likelihood estimation method, which is the following formula:

[0075]

[0076] Where, is the Jacobian determinant of J(μ,σ), μ is the expected value parameter, σ is the hyperparameter; I is the unit matrix, the isotropic Gaussian distribution (spherical Gaussian distribution) refers to a multidimensional Gaussian distribution with the same variance in all directions, and the covariance is a positive real number multiplied by the identity matrix.

[0077] Next, update the expected value parameter μ by the following formula:

[0078]

[0079] Where α is an empirical coefficient, I is the identity matrix, μ is the expected value parameter, and σ is the hyperparameter.

[0080] The determining whether the expected value parameter update is completed includes:

[0081] Determine whether the expected value parameter update is completed based on the adversarial loss and the preset number of updated mathematical expectation value parameters;

[0082] If the adversarial loss L(x') = 0, it is determined that the mathematical expectation parameter μ has completed updating, and the next step S5 is performed;

[0083] If the adversarial loss L(x') ≠ 0 and the number of times of the preset mathematical expectation parameter is not reached, it is determined that the update of the mathematical expectation parameter μ is not completed, and the video data is attacked again, that is, returning to step S2;

[0084] If the adversarial loss L(x')≠0 and the number of times reaches the mathematical expectation parameter, it is determined that the mathematical expectation parameter μ has completed the update and the next step S5 is performed.

[0085] S5: Generate adversarial samples based on the standardized flow model and expected value parameters.

[0086] In the process of generating adversarial samples, the normalized flow model is used to generate adversarial samples x' from random vector z, where the random vector z obeys z~N(z|μ,σ 2 I), the adversarial sample formula is:

[0087] x'=(projs(f(z)),z~N(z|μ,σ 2 I)

[0088] Where I is the identity matrix, μ is the expected value parameter, σ is the hyperparameter, and f(z) is the normalized flow generation model.

[0089] By determining the expected value parameter μ in step S4, adversarial samples of enhanced data can be accurately output, thereby improving the robustness of the model.

[0090] S6: Input adversarial examples into contrastive learning.

[0091] In a standard contrastive learning model, such as Figure 3 In contrastive learning, adding adversarial examples as positive samples to the training process is equivalent to using adversarial attacks as a form of data augmentation.

[0092] In contrastive learning, there are no correct labels, only positive and negative samples. Therefore, the C&W loss cannot be directly used as the objective function for adversarial examples. Contrastive learning uses cosine distance to represent the similarity between samples. In the absence of an attack, the cosine distance between positive samples should be much smaller than the cosine distance between positive and negative samples. When constructing adversarial examples, if the cosine distance between positive samples is less than the minimum of the cosine distances between positive and negative samples, it is considered a "misclassification," indicating a successful attack.

[0093] Embodiments of the present invention provide a self-supervised behavior recognition method based on adversarial training. This method attacks samples using a normalized flow adversarial attack and incorporates the attacked samples into contrastive learning training as positive samples, thereby increasing the number of positive samples. Furthermore, because adversarial samples only alter a few pixels and leave semantic information unchanged, while increasing the distance between positive samples, adversarial attacks can be used as a highly effective data augmentation method, a method for constructing difficult examples.

[0094] Figure 4 A schematic diagram of the structure of a self-supervised behavior recognition system in an embodiment of the present invention is shown. This self-supervised behavior recognition system is used in the aforementioned adversarial training-based self-supervised behavior recognition method. The self-supervised behavior recognition system includes: an extraction module 201 that extracts video data and inputs it into a standardized flow model; an attack module 202 that performs attack processing on the video data to generate an adversarial video; a technique module 203 that calculates the adversarial loss between the video data and the adversarial video in a learning mode; an update module 204 that updates the expected value parameters of the standardized flow model based on the adversarial loss and determines whether the update is complete; an adversarial generation module 205 that generates adversarial samples based on the standardized flow model and the expected value parameters; and an attack determination module 207 that inputs the adversarial samples into comparative learning.

[0095] Embodiments of the present invention provide a self-supervised behavior recognition method and system based on adversarial training. This method attacks samples using an adversarial attack based on a normalized flow. These samples are then treated as positive samples and incorporated into contrastive learning training, thereby increasing the number of positive samples. Furthermore, because adversarial samples only alter a few pixels and leave semantic information unchanged, they increase the distance between positive samples. Therefore, adversarial attacks can be used as an efficient data augmentation method and a method for constructing difficult examples. Furthermore, because the images generated by the normalized flow have a distribution similar to that of the original image, the image generated for a specific frame of a video in this method can be considered the video background. This background addition makes the image more likely to be used as a secondary sample, thereby reducing the impact of background bias in the dataset during training on model performance. Furthermore, training using adversarial attacks can improve model robustness. Training and testing using the behavior recognition datasets UCF-101 and HMDB-51 show that adversarially trained network models achieve higher recognition rates than non-adversarially trained network models. Furthermore, when subjected to the corresponding adversarial attacks, the adversarially trained network exhibits a lower misclassification probability, demonstrating the enhanced robustness of the adversarially trained model.

[0096] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk or an optical disk, etc.

[0097] In addition, the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A self-supervised behavior recognition method based on adversarial training, characterized in that: The self-supervised behavior recognition method based on adversarial training includes: Extract video data and input it into the standardized flow model; Perform attack processing on video data to obtain adversarial video; Calculate the adversarial loss of video data and adversarial video in learning mode; Update the expected value parameters of the normalized flow model based on the adversarial loss and determine whether the update is complete; If so, generate adversarial samples based on the normalized flow model with expected value parameters; Inputting the adversarial sample into contrastive learning; extracting video data includes: Extract a batch of video data, denoted as x i , input the video data into the standardized flow model and set the number of times to update the mathematical expectation parameter; Inputting adversarial samples into contrastive learning includes: In the standard contrastive learning model, a third branch is added, which consists of constructed adversarial samples, and the adversarial samples are used as positive samples for data enhancement.

2. The self-supervised behavior recognition method based on adversarial training according to claim 1, characterized in that The attack processing on the video data includes: A frame is randomly extracted from the video data, pixels are generated through the standardized flow model, and added to each frame of the entire video to obtain an adversarial video.

3. The self-supervised behavior recognition method based on adversarial training according to claim 1, characterized in that The calculation of the adversarial loss of the video data and the adversarial sample in the learning mode includes: The formula for calculating the adversarial loss is: in represents the input sample that has been attacked, C() represents the classifier, Represents the output score of the classifier for the yth item, which is the correct item, Represents the maximum score among all classes.

4. The self-supervised behavior recognition method based on adversarial training according to claim 1, characterized in that The method of updating the expected value parameters of the normalized flow model based on the adversarial loss includes: Use natural selection strategy to repeatedly update expected value parameters; According to the lazy man's theorem, the loss function formula of the natural selection strategy in this scenario is expressed as: Where, represents the loss in the natural selection strategy, In order to meet the requirement of invisibility of adversarial attack, the generated adversarial samples are projected into the corresponding space; the right side of the loss function formula of the natural selection strategy represents when the z vector obeys Under the condition of distribution, k z vectors are input into the standardized flow model f() and then added The mathematical expectation of the loss L of k obtained in the classifier is the adversarial sample generated by the restriction; Using the maximum likelihood estimation method for the above formula, we can find The Jacobian determinant of : Where, for The Jacobian determinant of , I is the identity matrix, is the expected value parameter, is a hyperparameter; Next, update the expected value parameter μ by the following formula: Where, is an empirical coefficient, I is the unit matrix, is the expected value parameter, is a hyperparameter.

5. The self-supervised behavior recognition method based on adversarial training according to claim 3, characterized in that The determining whether the update is completed includes: Determine whether the expected value parameter update is completed based on the adversarial loss and the preset number of updated mathematical expectation value parameters; If you fight against loss When , it is determined that the mathematical expectation parameter μ has completed the update and the next step is carried out; If you fight against loss , and if the preset number of times of updating the mathematical expectation value parameter is not reached, it is determined that the update of the mathematical expectation value parameter μ is not completed, and the video data is attacked again; If you fight against loss , and when the number of times of updating the mathematical expectation parameter is reached, it is determined that the mathematical expectation parameter μ has completed the update and the next step is performed.

6. The self-supervised behavior recognition method based on adversarial training according to claim 4, characterized in that The method of generating adversarial samples based on the standardized flow model and the expected value parameters includes: In the process of generating adversarial samples, the normalized flow model is used to generate adversarial samples from random vectors z. , where the random vector z obeys , the adversarial sample formula is: ； Where I is the identity matrix, is the expected value parameter, is a hyperparameter.

7. The self-supervised behavior recognition method based on adversarial training according to claim 1, characterized in that In the contrastive learning mode, cosine distance is used to represent the similarity between samples. If the cosine distance between positive samples is less than the minimum value of the cosine distance between positive and negative samples, the attack is considered to be successful.

8. A self-supervised behavior recognition system based on adversarial training, characterized in that The self-supervised behavior recognition system is used in the self-supervised behavior recognition method based on adversarial training according to any one of claims 1 to 7, and the self-supervised behavior recognition system includes: Extraction module: extracts video data and inputs it into the standardized flow model; Attack module: performs attack processing on video data to obtain adversarial video; Technical module: Calculate the adversarial loss of video data and adversarial videos in learning mode; Update module: updates the expected value parameters of the normalized flow model based on the adversarial loss and determines whether the update is completed; Adversarial generation module: Generates adversarial samples based on the normalized flow model and expected value parameters; Attack judgment module: input adversarial samples into contrastive learning.

Images