Distributed out-generalization method and system for convolutional neural networks

By performing style randomization and adversarial training on convolutional neural networks, the problem of insufficient distribution generalization ability in the existing technology is solved, and the effective generalization of multiple distribution drifts under small overhead is achieved, which improves the model's resistance Interference capability and scalability.

CN118446885BActive Publication Date: 2024-11-22WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410660023.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-27
Publication Date
2024-11-22
Estimated Expiration
2044-05-27

AI Technical Summary

Technical Problem

The existing external distribution generalization methods show performance attenuation when facing data distribution drift, and most methods are only effective for a single type of distribution drift, which cannot stably improve the generalization ability of the model, and have a large resource overhead.

Method used

A method of distribution generalization for convolutional neural networks was designed. Through style randomization and adversarial training, data features were extracted and style randomized for convolutional neural network structures were performed. Combined with FGSM adversarial training algorithm, decision-making was optimized. Boundaries, reduce the risk of overfitting, suitable for complex real-life tasks and real data sets.

Benefits of technology

Under smaller time and resource overhead, it is effective to generalize p(x) and p(y|x) distribution drifts simultaneously, which is scalable, reduces the computational overhead of adversarial sample generation and improves the model's resistance Interference capability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118446885B_ABST
    Figure CN118446885B_ABST
Patent Text Reader

Abstract

The present invention discloses a distributed out-generalization method and system for convolutional neural networks, which include two core technologies: first, randomizing the style of image features extracted from the low-level network structure of the convolutional neural network, generating a new style based on a random weight mixing algorithm, and using a style transfer technology to enhance the style diversity of the original data, thereby reducing the correlation between the image style and the image content, and prompting the model to focus on content information that has a stronger causal relationship with the output label; second, performing adversarial training on the entire training process of the neural network model, optimizing the search range of the single-step gradient ascent algorithm for the optimal solution in the constraint space based on random initial noise, and using the single-step gradient ascent algorithm to generate adversarial samples, so that the model actively learns the deep features of the image and the correct classification labels of the perturbed samples, and reducing the risk of overfitting the model to the training data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of deep neural networks in artificial intelligence, and in particular to a distributed generalization method and system for convolutional neural networks. Background Art

[0002] In recent years, with the rapid progress of computer hardware and software technology and the continuous accumulation of massive data, artificial intelligence and its related technologies have also ushered in unprecedented vigorous development. According to statistics from Xinhua News Agency, the scale of my country's core artificial intelligence industry will reach 500 billion yuan in 2023, the number of core enterprises will exceed 4,300, and innovative achievements will continue to emerge. It can be foreseen that in the future, various fields including medical care, transportation, finance, and manufacturing will usher in digital and intelligent transformation and upgrading.

[0003] The achievements of artificial intelligence are inseparable from the rapid development of deep neural networks (DNN). As a powerful machine learning model, deep neural networks have played an important role in computer vision, natural language processing, intelligent speech recognition and other fields. Among them, convolutional neural networks (CNN) have shown particularly obvious advantages in image recognition and classification tasks, which has greatly promoted the application of computer vision in many real-world tasks, such as financial fraud detection, anomaly detection, intelligent driving, medical diagnosis, etc. These high-risk tasks have put forward higher requirements on the accuracy and reliability of deep neural networks on real data.

[0004] Although deep neural networks have excellent performance in many experimental scenarios, they are vulnerable to data with different distributions in real-world tasks. Usually, the training data of deep neural network models are sampled from certain specific data distributions (also known as domains), and the test data and training data are always assumed to be independent and identically distributed in experiments. This leads to obvious performance degradation of the final neural network model when facing data with different distributions from the training set. The academic community calls this type of data Out-of-Distribution (OOD) data. If x represents the input data, y represents the label corresponding to the input data, and the probability distribution is p(·), then the formation of OOD data is mainly caused by the distribution drift of p(x) and the distribution drift of p(y|x). Since the data in real-world scenarios are complex and diverse, and various types of distribution drifts are widely present, it is difficult to collect or predict all possible test data in advance. Therefore, training a model with strong generalization ability for OOD data on limited data has become an important research topic of deep neural networks. The solution to this topic is called the out-of-distribution generalization method.

[0005] Existing out-of-distribution generalization methods have very obvious limitations in performance and efficiency. For example, some methods based on adversarial networks or generative networks require additional training processes; some adversarial training methods based on multi-step adversarial attacks such as the PGD algorithm require large resource overhead for multiple gradient calculations; most methods based on invariance learning rely on some unrealistic assumptions to achieve the theoretical effects of the algorithm; more importantly, most out-of-distribution generalization methods are only effective for one of the distribution drifts p(x) and p(y|x), and cannot stably improve the model's out-of-distribution generalization ability on most real OOD data formed by the combined effect of two distribution drifts, and may even make the model performance decay more seriously. Therefore, how to design a stable, general and efficient out-of-distribution generalization method has become a problem that needs to be solved urgently. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention designs a distributed external generalization method and system for convolutional neural networks, which realizes a lightweight distributed external generalization method that is effective for both the distribution drift of p(x) and the distribution drift of p(y|x) with relatively small time and resource overhead. The method is suitable for various complex real-world tasks and real data sets, and is scalable to different convolutional neural networks.

[0007] The invention designs a distribution out-generalization method for convolutional neural networks, which performs style randomization on data features extracted from low-level network structures in convolutional neural networks; conducts adversarial training on the entire training process of the convolutional neural network model, allowing the model to actively learn the deep features of disturbed samples and their correct classification labels, thereby optimizing the decision boundary and reducing the risk of overfitting the training data;

[0008] Style randomization uses a data enhancement algorithm based on style transfer to extract style features s from the current data features of the original image x; randomly select another image x′ in the training data as a reference image, and extract style features s′ from the current data features of the reference image; randomly weight the style s of the original image and the style s′ of the reference image to generate a new style feature s new ; Add new style features new Migrate to the original image x to randomize the style of the original image.

[0009] Furthermore, the low-level network structure refers to a position structure in a convolutional neural network that tends to learn style features in an image.

[0010] Furthermore, the adversarial training adopts an adversarial training algorithm based on FGSM, which specifically includes the following steps:

[0011] Randomly sample the noise η from the uniform distribution Uniform(-α,α), add the noise η to the original image x, and get the noise-enhanced sample x aug ; Use the FGSM algorithm to calculate the model loss function for x aug The gradient direction of x and a small perturbation in that direction is added to the sample x aug Generate adversarial examples The adversarial sample As training data, according to the model Classification results Calculate the loss function value and optimize and update the model parameters through back propagation.

[0012] Furthermore, the specific process of the style randomization includes the following steps:

[0013] C01: For each input tensor x in a batch, calculate its sample-level mean and standard deviation Use the mean-standard deviation pair to represent the style feature s = (μ(x), σ(x)) of sample x;

[0014] C02: Randomly select another sample x′ in the batch in the training data as a reference sample and calculate its sample-level mean and standard deviation The style feature s′=(μ(x′),σ(x′)) of sample x′ is represented by the mean-standard deviation pair;

[0015] C03: The style s of the original sample and the style s′ of the reference sample are weighted mixed using a hyperparameter λ to obtain a new mean μ new =λμ(x)+(1-λ)μ(x′) and new variance σ new =λσ(x)+(1-λ)σ(x′), the new mean-standard deviation pair can represent a new style feature s new =(μ new ,σ new );

[0016] C04: New style features new Migrate to the original image x and get a sample with randomized style

[0017] Based on the same inventive concept, the present invention also designs a distributed out-generalization system for convolutional neural networks, including a style randomization module, which performs style randomization on the data features extracted from the low-level network structure of the convolutional neural network, and uses a data enhancement algorithm based on style transfer to extract style features s from the current data features of the original image x; randomly selects another image x′ as a reference image in the training data, and extracts style features s′ from the current data features of the reference image; compares the style s of the original image with the style of the reference image Perform random weight mixing to generate new style features The new style features Migrate to the original image x to randomize the style of the original image, weaken the correlation between image style and image content, and make the model pay more attention to the content characteristics of the image;

[0018] The adversarial training module performs adversarial training on the entire training process of the convolutional neural network model, using the FGSM adversarial training algorithm with random initial perturbation values ​​to allow the model to actively learn the deep features of perturbed samples and their correct classification labels, thereby optimizing the decision boundary and reducing the risk of overfitting the training data.

[0019] Based on the same inventive concept, the present invention also provides an electronic device, including:

[0020] one or more processors;

[0021] A storage device for storing one or more programs;

[0022] When one or more programs are executed by the one or more processors, the one or more processors implement a distributed out-of-generalization method for convolutional neural networks.

[0023] Based on the same inventive concept, the present invention also designs a computer-readable medium on which a computer program is stored. When the program is executed by a processor, a distributed out-generalization method for convolutional neural networks is implemented.

[0024] Compared with the prior art, the advantages of the present invention are:

[0025] It can work on the distribution drift of p(x) and the distribution drift of p(y|x) at the same time with a relatively small computational cost, and is scalable to various convolutional neural networks. Specifically, the implementation of the present invention does not require the introduction of additional network modules, and can be flexibly deployed without changing the original structure of the neural network. At the same time, in the adversarial training algorithm proposed by the present invention, only one gradient calculation is required for each adversarial sample generated, so it can work on the distribution drift of p(x) and the distribution drift of p(y|x) at the same time with a relatively small computational cost.

[0026] In response to the distribution drift of p(x), the present invention designs a new style randomization algorithm, which generates a new style and injects the original training data during the training process, thereby weakening the correlation between image content and image style, and prompting the model to pay more attention to the content characteristics of the image; in response to the distribution drift of p(y|x), the present invention designs a new adversarial training algorithm, which uses random initial noise to optimize the search range of the single-step gradient ascent algorithm for the optimal solution in the constraint space, thereby creating more diverse adversarial samples for adversarial training and improving the model's anti-interference ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the framework of the distributed out-generalization method for convolutional neural networks in the embodiment.

[0028] Figure 2 1 is a schematic diagram of the implementation flow of the distributed out-generalization method in the embodiment.

[0029] Figure 3 Schematic diagram of the framework of the distributed out-of-generalization method for ResNet-18 in the embodiment. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described below in conjunction with the accompanying drawings and embodiments.

[0031] Embodiment 1

[0032] The present invention considers the following out-of-distribution generalization problem: In a supervised learning scenario, X is the input data, Y is the label corresponding to the input data, and a neural network model F:X→Y is in the data set The model needs to be applied to another dataset. Dataset The data in The data in the data set has a distribution drift. If the probability distribution is denoted by p(·), this distribution drift includes both the drift of P(x) and the drift of P(y|x). The goal of this invention is to minimize the prediction error of the model on these OOD data, that is, to find a prediction model f∈F that can minimize the empirical risk in the data set. If the input data is x, the model's predicted label for the input data is f(x), the true label of the input data is y, and the model's loss function is L, then the goal can be expressed as:

[0033]

[0034] in Indicates expected value.

[0035] The present invention will be further described below in conjunction with specific implementations and with reference to the accompanying drawings.

[0036] The distribution out-generalization method of the present invention is mainly aimed at convolutional neural networks. The network structure of convolutional neural networks is as follows: Figure 1 As shown in the figure, it usually includes a preprocessing module, multiple convolution modules and a classifier module, wherein the preprocessing module is mainly responsible for converting the original image into a tensor, the convolution module is mainly responsible for extracting features from the tensor, and the classifier module is mainly responsible for judging the image label based on the extracted features. The main difference between different convolutional neural networks lies in the different composition of the convolution modules in the network. For example, for AlexNet, each convolution module contains a convolution layer and a maximum pooling layer; for VGG, each convolution module contains multiple convolution layers and a maximum pooling layer; for ResNet, the convolution module is a residual block, and each residual block is formed by combining multiple convolution layers under a residual structure.

[0037] The present invention fully considers and utilizes the processing characteristics of convolutional neural networks on image features: on the one hand, for the convolution module of the low-level part of the convolutional neural network, the model is more inclined to extract the style features of the image, so the style randomization algorithm can be implemented on the extracted data features in the low-level network; on the other hand, the convolutional neural network processes the image in a process in which the feature extractor mines the image content features and then the classifier outputs the image label, so the entire training process of the model can be subjected to adversarial training.

[0038] Specifically, the implementation of the distribution out-generalization method of the present invention in a convolutional neural network with N convolutional modules is as follows: Figure 2 As shown, in each round of iteration, firstly, an image sample in the training set is selected, processed and encoded by the preprocessing module in the neural network to obtain the tensor x; then, a disturbance is added to the tensor x to obtain the tensor The perturbation includes a random noise randomly sampled from a specific distribution and an adversarial noise of the model's loss function on the current input tensor in the gradient direction; then let the model Perform adversarial training to make the tensor After each convolution module, the input tensor is randomized to reduce the correlation between the content features and style features in the image and the correlation between the style features and the image labels, so that the model can focus more on learning the causal relationship between the image content features and the image labels. Finally, let the tensor Pass through the Nth convolution module and classifier in sequence to get the predicted label Then, the gradient of the loss to the model parameters is calculated through back propagation, and the optimization algorithm is used to update the model parameters.

[0039] The present invention is described in detail below using a commonly used convolutional neural network, ResNet-18, as an example.

[0040] The schematic diagram of the distributed generalization framework for ResNet-18 in the embodiment of the present invention is as follows Figure 3 As shown in the figure, the network structure of ResNet-18 contains four residual blocks. The first three residual blocks focus more on the style features of the image, and the last residual block focuses more on the content features of the image because it is connected to the classifier. Therefore, the out-of-distribution generalization method for ResNet-18 mainly includes the following steps:

[0041] Step 1: In each round of iteration, obtain the tensor x of the input image, add random noise and perturbations in the gradient direction of the loss function to x, and generate adversarial samples of x

[0042] Specifically, adversarial examples The generation algorithm is shown in Table 1, which includes four steps: S01: randomly sample a noise η from the uniform distribution Uniform(-α,α); S02: add the noise η to the tensor x of the original sample to obtain the noise-enhanced sample x aug =x+η; S03: Get sample x aug Output results under the current model Calculate the model loss function for x aug The gradient of , use the sign function sign(·) to get the direction of the gradient, and multiply the direction by a step size ∈ to get a small perturbation S04: Add perturbation δ to sample x aug On the above, we get the adversarial sample

[0043] Table 1

[0044]

[0045] Step 2: Instead of the original sample, continue forward propagation in ResNet-18, passing through the convolutional layers in sequence Normalization layer Activate Layer and pooling layer

[0046] Step 3: Let Continue forward propagation in ResNet-18 to reach the first residual block, and get the output of the first residual block right The style features are randomized to obtain samples Reduce the correlation between the style characteristics and content characteristics of the sample.

[0047] Specifically, if the original tensor of an input sample is recorded as Where B represents the size of an input batch, C represents the number of channels of an image, H and W represent the height and width of an image respectively. The style randomization algorithm is shown in Table 2 and includes four steps: C01: For each input tensor x in a batch, calculate its sample-level mean and standard deviation Use the mean-standard deviation pair to represent the style feature s = (μ(x), σ(x)) of sample x; C02: Randomly select another sample x′ in the batch in the training data as a reference sample and calculate its sample-level mean and standard deviation The style feature s′ of sample x′ is represented by the mean-standard deviation pair s′=(μ(x′),σ(x′)); C03: The style s of the original sample and the style s′ of the reference sample are weighted mixed using a hyperparameter λ to obtain a new mean μ new =λμ(x)+(1-λ)μ(x′) and new variance σ new =λσ(x)+(1-λ)σ(x′), the new mean-standard deviation pair can represent a new style feature s new =(μ new ,σ new ), where the value range of the hyperparameter λ is between [0,1] and is usually set to 0.5. Its value can be manually adjusted according to the actual situation in the experiment; C04: The new style feature s new Migrate to the original image x and get a sample with randomized style

[0048] Table 2

[0049]

[0050] Step 4: Let Continue forward propagation in ResNet-18 to reach the second residual block, repeat the steps in Step 3, and obtain the sample that has passed the second residual block and has been style randomized.

[0051] Step 5: Let Continue forward propagation in ResNet-18 to reach the third residual block, repeat the steps in Step 3, and obtain the sample that has passed the third residual block and has been style randomized.

[0052] Step 6: Let Continue forward propagation in ResNet-18, passing through the fourth residual block in turn Pooling Layer Then it is passed into the classifier to get the classification result

[0053] Step 7: Based on the classification results Calculate the loss function value and optimize and update the model parameters through back propagation.

[0054] Embodiment 2

[0055] Based on the same inventive concept, the present invention also designs a distributed out-generalization system for convolutional neural networks, including a style randomization module, which performs style randomization on data features extracted from the low-level network structure of the convolutional neural network, and uses a data enhancement algorithm based on style transfer to extract style features s from the current data features of the original image x; randomly selects another image x′ as a reference image in the training data, and extracts style features s′ from the current data features of the reference image; randomly weights the style s of the original image and the style s′ of the reference image to generate a new style feature s new ; Add new style features new Migrate to the original image x to randomize the style of the original image;

[0056] Adversarial training module: conducts adversarial training on the entire training process of the convolutional neural network model, allowing the model to actively learn the deep features of perturbed samples and their correct classification labels, thereby optimizing the decision boundary and reducing the risk of overfitting the training data.

[0057] Since the system introduced in the second embodiment of the present invention is a system adopted for implementing the distributed generalization method for convolutional neural networks in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, technical personnel in this field can understand the specific structure and deformation of the electronic device, so it will not be repeated here.

[0058] Embodiment 3

[0059] Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; when one or more programs are executed by the one or more processors, the one or more processors implement the method described in Example 1.

[0060] Since the device introduced in the third embodiment of the present invention is an electronic device used to implement the distributed generalization method for convolutional neural networks in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the electronic device, so it is not repeated here. All electronic devices used in a method in the embodiment of the present invention belong to the scope of protection of the present invention.

[0061] Embodiment 4

[0062] Based on the same inventive concept, the present invention further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method described in the first embodiment is implemented.

[0063] Since the device introduced in the fourth embodiment of the present invention is a computer-readable medium used to implement the distributed generalization method for convolutional neural networks in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the electronic device, so it is not repeated here. All electronic devices used in a method in the embodiment of the present invention belong to the scope of protection of the present invention.

[0064] The above-described embodiment only expresses a specific implementation of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. A distributed out-generalization method for convolutional neural networks, characterized by: The style of the data features extracted from the low-level network structure of the convolutional neural network is randomized, and adversarial training is performed on the entire training process of the convolutional neural network model, so that the model can actively learn the deep features of the perturbed samples and their correct classification labels, optimize the decision boundary, and reduce the risk of overfitting the training data; Style randomization uses a data augmentation algorithm based on style transfer. For each input tensor x in a batch, its sample-level mean and standard deviation are calculated, and the mean-standard deviation pair is used to represent the extracted style feature s of sample x. In the training data, randomly select another sample x' in the batch as a reference sample, calculate its sample-level mean and standard deviation, and use the mean-standard deviation pair to represent the style feature s' of sample x'; randomly weight the style s of the original image and the style s' of the reference image to obtain a new mean and a new variance. The new mean-standard deviation pair represents a new style feature s new ; Add new style features new Migrate to the original image x to randomize the style of the original image; Adversarial training uses a new adversarial training algorithm based on FGSM, which includes the following steps: Randomly sample the noise η from the uniform distribution Uniform(-α,α), add the noise η to the original image x, and get the noise-enhanced sample x aug ; Use the FGSM algorithm to calculate the model loss function for x aug The gradient of x and a small perturbation in the direction of the gradient is added to the sample x aug Generate adversarial examples The adversarial sample As training data, according to the model Classification results Calculate the loss function value and optimize and update the model parameters through back propagation.

2. The distributed out-generalization method for convolutional neural networks according to claim 1, characterized in that: The low-level network structure refers to a position structure in a convolutional neural network that tends to learn style features in an image.

3. A distributed out-generalization system for convolutional neural networks, characterized by: It includes a style randomization module. The style randomization adopts a data enhancement algorithm based on style transfer. For each input tensor x in a batch, its sample-level mean and standard deviation are calculated, and the mean-standard deviation pair is used to represent the extracted style feature s of sample x. In the training data, randomly select another sample x' in the batch as a reference sample, calculate its sample-level mean and standard deviation, and use the mean-standard deviation pair to represent the style feature s' of sample x'; randomly weight the style s of the original image and the style s' of the reference image to obtain a new mean and a new variance. The new mean-standard deviation pair represents a new style feature s new ; Add new style features new Migrate to the original image x to randomize the style of the original image; The adversarial training module conducts adversarial training on the entire training process of the convolutional neural network model. It uses the FGSM adversarial training algorithm with random initial perturbation values ​​to allow the model to actively learn the deep features of the perturbed samples and their correct classification labels, thereby optimizing the decision boundary and reducing the risk of overfitting the training data. Adversarial training uses a new adversarial training algorithm based on FGSM, which includes the following steps: Randomly sample the noise η from the uniform distribution Uniform(-α,α), add the noise η to the original image x, and get the noise-enhanced sample x aug ; Use the FGSM algorithm to calculate the model loss function for x aug The gradient of x and a small perturbation in the direction of the gradient is added to the sample x aug Generate adversarial examples The adversarial sample As training data, according to the model Classification results Calculate the loss function value and optimize and update the model parameters through back propagation.

4. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the distributed out-generalization method as described in any one of claims 1-2.

5. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed out-generalization method as described in any one of claims 1-2 is implemented.