An image recognition method based on distributed machine learning and privacy protection technology

Abstract

The application discloses an image recognition method based on distributed machine learning and privacy protection technology, and the method is as follows: a saliency mapping attack method based on Jacobian matrix is adopted to generate an adversarial sample; reversible down-sampling and up-sampling layers are applied to a client and a server respectively, four input tensors are obtained according to color channels, and a convolution function is used to combine the input tensors to obtain a combined tensor; a Lipschitz continuous gradient function is used to solve a model obtained by a proximal gradient algorithm, robust principal component analysis and global optimization analysis to obtain a restored image; convolution and strategy are used to realize synthesis of an activation map, and linear summation is performed on the activation map to obtain a combined image; an attack image, the combined image and the restored image are recognized respectively; an adaptive weighted average method is used to sum the recognition results of the three images to obtain an image recognition result. The application can protect image privacy and improve the recognition accuracy of an image recognition model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image recognition, and particularly relates to an image recognition method based on distributed machine learning and privacy protection technology. BACKGROUND

[0002] In the past decade, artificial intelligence technology, especially deep learning network, has developed rapidly; the proposal and development of the Internet of Things have made distributed machine learning widely adopted, and the combination of the two has made technology enter a new era, greatly facilitating production and life, and penetrating into all aspects of life. However, the security problem of deep learning network has become the biggest obstacle on its development road: an unsafe learning network not only may cause user sensitive information to be stolen, but also may cause fatal misidentification results. Such insecurity and the results it may cause make more users give up using deep learning network.

[0003] At the same time, higher recognition accuracy is also the benefit expectation of users when using deep learning network.

[0004] Therefore, the application proposes an image recognition method based on distributed machine learning and privacy protection technology, which uses attack and defense technology in neural network, processes information at the user end, prevents privacy leakage in the transmission process, restores information at the server end, and achieves higher recognition accuracy. While protecting privacy, the efficiency of deep learning network is improved. SUMMARY

[0005] The application proposes an image recognition method based on distributed machine learning and privacy protection technology to solve the problem of privacy leakage and improve the recognition accuracy of deep learning network.

[0006] To achieve the above application purposes, the technical scheme adopted by the application is:

[0007] An image recognition method based on distributed machine learning and privacy protection technology, the method comprising the following steps:

[0008] Step 1: generation of adversarial samples:

[0009] The saliency mapping attack method based on Jacobian matrix calculates the influence of each pixel in the image on the predicted label category:

[0010] When the image x corresponds to the label y, the relationship between the pixel point x i and y in the image x is:

[0011]

[0012] When the image x does not correspond to the label y', the relationship between the pixel point x i and y' in the image x is:

[0013]

[0014] To generate the adversarial sample, the result of the saliency map can be changed to change the connection between the pixel and the label. Changing the value of the pixel in the image enhances the connection between a certain pixel and the non-corresponding label, and weakens the connection between it and the corresponding label, so as to obtain an adversarial sample without a specific attack target, and the adversarial sample is used as an attack image:

[0015]

[0016]

[0017] Step 2: Sampling of image samples: reversible up-sampling and down-sampling layers are applied to the client and server respectively, and the image samples are sampled, which can provide prior information for the generation of adversarial samples, be used for the design of the convolution layer in the down-sampling layer, increase the receiving range while maintaining the network depth, improve the training speed, and also reduce the artificial factors that may affect the visual quality. Since the four input tensors obtained after sampling need to be merged to obtain the restored image in step 4, reversible sampling layers are used here so that the sub-images after sampling can be merged.

[0018] Step 3: Obtain adversarial sample tensor: in the down-sampling layer, generate four input tensors i x j x (4ch+1) corresponding to an i x j image according to the saliency map attack method of the Jacobian matrix, wherein i is the number of pixels in the length direction of the image, j is the number of pixels in the width direction of the image, and ch is the number of color channels of the image.

[0019] Step 4: Obtain the restored image: merge the input tensors in step 3 using the convolution function g(x) to obtain the merged tensor perform inverse mapping of the saliency map and input it into the solving model to obtain the restored image; since the mapping relationship has certain randomness, and in order to improve the restoration accuracy, the proximal gradient algorithm, robust principal component analysis (RPCA) and global optimization analysis are used to obtain the solving model

[0020]

[0021] μ is a relaxation parameter greater than 0, which is selected according to experience in a specific model; H represents the Hilbert space, wherein g(x) is a convolution function, and f(x) is a smooth convex function:

[0022]

[0023] ​f(x) can be solved using a Lipschitz continuous gradient function:

[0024]

[0025] where x i represents a specific image in the image set, represents the output value obtained after inverse mapping thereof;

[0026] After solving f(x), f(x), g(x) and μ are substituted into the first equation to solve the model.

[0027] Step 5: Obtain the merged image: under different color channels, the merged tensor is obtained by convolution; based on the convolution and method, a set of filters and bias are used to realize the convolution to connect the activation maps obtained under different color channels; in the process of convolution, a weighted combination of activation maps is generated to realize dimension reduction; linear summation is performed on the activation maps to be merged to obtain the merged image;

[0028] Step 6: Recognition of different types of images: a simple convolutional neural network with 7 convolutional layers and 2 fully connected (FC) layers is used to recognize the attack image, the restored image and the merged image respectively to obtain the recognition results corresponding to the three groups of images; in the simple convolutional neural network, each convolutional layer has a kernel size of 3x3 and 64 channels, and a segmentation layer is set after the second convolutional layer;

[0029] Step 7: Merging of recognition results: the maximum average value of the specific class label in each group of image recognition results is calculated, and then the adaptive weighted average method is used to weighted sum the image recognition results of the three types to obtain the final recognition result.

[0030] L a , L f , L r represent the classification parameters obtained by the attack image, the merged image and the restored image respectively, and the adaptive weighted average method is used to weighted sum the image recognition results of the three types, which is specifically:

[0031] L awa = γ*L r + ρ*L f + (1-ρ-γ)*L a

[0032] ε γ , ε β , ε ρ represent L a , L f , L rThe proportion;

[0033] The calculation formula of gamma and rho is as follows:

[0034]

[0035]

[0036] Wherein, And Respectively represent the maximum average value of the specific class label classification parameter in the attack image, the merged image and the recovery image.

[0037] Further, the present application includes the following preferred schemes:

[0038] In step 1, the input sample is subjected to an untargeted adversarial attack, which belongs to a white-box attack. In the case of known model parameters, the confidence of the model in predicting the true label is reduced by modifying the corresponding relationship to eliminate the dependence on this target class, thereby achieving the purpose of adversarial attack, i.e. false recognition.

[0039] In step 1, the generated adversarial sample not only effectively resists privacy threats caused by data leakage in the machine learning process, but also serves as a prototype for the recovery image and the merged image, and the three together make the final result prediction.

[0040] In step 2, in the reversible down-sampling layer, the input image is reconstructed into a set of small sub-images, and the sub-images are subjected to fast inference and noise reduction according to the image noise level mapping. The application of hybrid orthogonal projection and estimation method in the convolution layer of the down-sampling layer can improve the generalization ability of the down-sampling layer under non-uniform and spatially variant noise levels, and enhance the adaptability to image noise, which is suitable for the input of adversarial samples.

[0041] At the same time, denoising the down-sampled sub-image can also effectively expand the receiving range, thereby obtaining a moderate depth of neural network.

[0042] In step 3, four input tensors are generated, i.e. the sample is divided into 4 parts for sampling and processing respectively, with the purpose of minimizing the computational overhead while not reducing the processing capacity of the model.

[0043] In step 4, the approximate inverse mapping of the adversarial sample generation mapping in step 1 is first obtained. Since the mapping has a certain randomness in the generation of adversarial samples, it cannot generate an absolutely accurate inverse mapping, but needs to generate the closest inverse mapping through a global optimization problem, so as to obtain the closest recovery image.

[0044] In step 5, there are many strategies for merging activation maps, the common ones are summation, convolution and convolution and three strategies. Existing research has proved that convolution and merging have better merging effect.

[0045] In step 7, there are generally three merging strategies: weighted average, adaptive weighted average, and meta-learning, which can all improve recognition accuracy. Meta-learning uses a shallow classification method to calculate probability, which is more suitable in complex calculation; weighted average is more suitable in the case of opposite spectrum and has the least calculation constraint, and adaptive weighted average is the best merging strategy after balancing recognition efficiency and calculation complexity.

[0046] The present application has the following beneficial technical effects:

[0047] 1. High resistance: the user end performs an adversarial attack on the input data, and the server end performs recovery, improving the resistance to error samples through the process of mapping and inverse mapping;

[0048] 2. Good privacy: pre-transmission adversarial attack can encrypt the data to a certain extent, and like the encryption feature of discrete learning itself, it prevents data leakage during transmission;

[0049] 3. Strong adaptability: the upper and lower sampling layers are added, the upper sampling layer divides and reconstructs the sample, and the lower sampling layer improves the generalization ability of the sample under non-uniform and spatial variation noise level, and has better robustness to sample noise;

[0050] 4. Good recovery effect: the model obtained by the proximal gradient algorithm, robustness principle component analysis (RPCA) and global optimization analysis is solved by using the Lipschitz continuous gradient function, and the recovered image is more accurate than other methods from the visual judgment.

[0051] 5. High recognition accuracy: the three input streams (attack image, merged image, and recovered image) are used for training, and the classification results are adaptively weighted and averaged to obtain more accurate recognition results. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 It is a flow chart of an image recognition method based on distributed machine learning and privacy protection technology. DETAILED DESCRIPTION

[0053] The present application will be further described and explained in conjunction with the accompanying drawings and examples. The technical features of each embodiment of the present application can be combined accordingly without conflict.

[0054] In step 1, generate an adversarial sample. Based on the significant mapping attack method of Jacobian matrix, the influence of each pixel in the image x on the predicted label class y is observed by significant mapping, and it is considered that when the image x corresponds to the label y, the pixel point xi The relationship with y is:

[0055]

[0056] When the image x does not correspond to the label y', the pixel x i The relationship with y' is:

[0057]

[0058] To generate an adversarial sample,

[0059] The relationship between the pixel and the label can be changed by changing the results of the saliency mapping. Changing the value of a pixel in the image enhances the relationship between the pixel and the non-corresponding label, and weakens the relationship between the pixel and the corresponding label, thereby obtaining an adversarial sample without a specific attack target, and using the adversarial sample as an attack image:

[0060]

[0061]

[0062] In step 2, sampling of the image sample is performed: reversible up-sampling and down-sampling layers are applied to the client and server respectively, and the image sample is sampled, which can provide prior information for the generation of the adversarial sample, be used for the design of the convolutional layer in the down-sampling layer, increase the receiving range while maintaining the network depth, improve the training speed, and reduce the artificial factors that may affect the visual quality. Since the four input tensors obtained after sampling need to be combined into a restored image in step 4, reversible sampling layers are used here so that the sub-images after sampling can be combined.

[0063] In step 3, the adversarial sample tensor is obtained: in the down-sampling layer, according to the saliency mapping attack method of the Jacobian matrix, four input tensors i×j×(4ch+1) corresponding to an i×j image are generated, where i is the number of pixels in the length direction of the image, j is the number of pixels in the width direction of the image, and ch is the number of color channels of the image;

[0064] In step 4, the restored image is obtained: the input tensors in step 3 are combined using the convolution function g(x) to obtain a combined tensor The combined tensor Inverse mapping of saliency mapping is input into the solving model, thereby obtaining the restored image; since the mapping relationship has a certain randomness, and in order to improve the restoration accuracy, the proximal gradient algorithm, robust principal component analysis (RPCA) and global optimization analysis are used to obtain the solving model

[0065]

[0066] μ is a relaxation parameter greater than 0, which is selected empirically in a specific model; H represents a Hilbert space, wherein f(x) is a smooth convex function:

[0067]

[0068] The f(x) can be solved using a Lipschitz continuous gradient function:

[0069]

[0070] wherein x i represents a specific image in the image set, represents an output value obtained after inverse mapping thereof;

[0071] After the solution of f(x) is completed, f(x), g(x) and μ are substituted into the solution model, so that the solution model is solved; and the original image is restored.

[0072] In step 5, a merged image is obtained: under different color channels, the merged tensor is obtained through convolution; based on the convolution and method, a set of filters and bias are used to realize the convolution to connect the activation maps obtained under different color channels; in the process of convolution, a weighted combination of the activation maps is generated to realize dimension reduction; linear summation is performed on the activation maps to be merged to obtain the merged image;

[0073] In step 6, different types of images are identified: a simple convolutional neural network with 7 convolutional layers and 2 fully connected (FC) layers is used to identify the attack image, the restored image and the merged image respectively to obtain the identification results corresponding to the three groups of images; in the simple convolutional neural network, each convolutional layer has a kernel size of 3x3 and 64 channels, and a segmentation layer is set after the second convolutional layer;

[0074] In step 7, the identification results are merged: the maximum average value of the specific class label in each group of image identification results is calculated, and then the adaptive weighted average method is used to perform weighted summation on the identification results of the three types of images to obtain the final identification result.

[0075] L a , L f , L r represent the classification parameters obtained by the attack image, the merged image and the restored image respectively, and the adaptive weighted average method is used to perform weighted summation on the identification results of the three types of images, specifically:

[0076] L awa = γ * L r + ρ * Lf + (1 - p - y) * L a

[0077] ε γ , ε β , ε ρ respectively represent the proportion of L a , L f , L r ;

[0078] The calculation formula of y, p is as follows:

[0079]

[0080]

[0081] wherein, and respectively represent the maximum average value of the specific class label classification parameter in the attack image, the merged image, and the recovery image.

[0082] The above examples are used to explain and illustrate the present application, but not to limit the present application, any modification and change made to the present application within the spirit and protection scope of the claims of the present application, all fall into the protection scope of the present application.

Claims

1. An image recognition method based on distributed machine learning and privacy protection technology, characterized in that, The method comprises the following steps: Step 1: generation of an adversarial sample: The Jacobian matrix-based significance mapping attack method calculates the influence of each pixel in the image on the predicted label category: When the image x corresponds to the label y, the relationship between the pixels x i and y is: When the image x does not correspond to the label y', the pixel x i has a relationship with y' as follows: Changing the value of the pixel in the image enhances the connection of the pixel with the non-corresponding label and weakens its connection with the corresponding label, thereby obtaining an adversarial sample without a specific attack target, and taking the adversarial sample as an attack image: Step 2: sampling of the image sample: reversible up-sampling and down-sampling layers are applied to the client and server respectively and are used to sample the image sample; Step 3: obtaining of the adversarial sample tensor: in the down-sampling layer, four input tensors i x j x (4ch + 1) corresponding to an i x j image are generated according to the Jacobian matrix-based significance mapping attack method, wherein i is the number of pixels in the length direction of the image, j is the number of pixels in the width direction of the image, and ch is the number of color channels of the image; Step 4: Obtain the restored image: merge the input tensor in step 3 by using the convolution function g(x) to obtain a merged tensor performing significance mapping on the merged tensor performing inverse mapping of the significance mapping and inputting it into a solving model to obtain the restored image; the solving model is obtained based on a proximal gradient algorithm, a robust principal component analysis and a global optimization analysis. Step 5: Obtain the merged image: under different color channels, the activation maps obtained by merging the tensor obtained by convolution; based on the convolution and method, a set of filters and bias are used to implement the convolution to connect the activation maps obtained under different color channels; linear summation is performed on the activation maps that need to be merged to obtain the merged image; Step 6: identification of different types of images: a simple convolutional neural network with 7 convolutional layers and 2 fully connected layers is used to identify the attack image, the restored image and the merged image, and three groups of identification results corresponding to the images are obtained; in the simple convolutional neural network, each convolutional layer has a kernel size of 3 x 3 and 64 channels, and a segmentation layer is arranged after the second convolutional layer; Step 7: merging of the identification results: the maximum average value of the specific class label in each group of image identification results is obtained, and then the adaptive weighted average method is used to perform weighted summation on the image identification results of the three types, thereby obtaining the final identification result.

2. The image recognition method based on distributed machine learning and privacy protection technology according to claim 1, wherein: The solving model in step 4 is specifically: Wherein, μ is a relaxation parameter greater than 0, which is selected according to experience in a specific model; H represents a Hilbert space, g(x) is a convolution function, and f(x) is a smooth convex function The Lipschitz continuous gradient function is used to solve f(x) wherein x i represents the ith image to be recognized, represents the output value obtained after inverse mapping x i .

3. The image recognition method based on distributed machine learning and privacy protection technology according to claim 1, wherein: In step 7, the adaptive weighted average method is used to perform weighted summation on the image identification results of the three types, and the specific calculation formula is as follows: L awa = γ * L r + ρ * L f + (1 - ρ - γ) * L a Wherein, L awa is the result of weighted sum of three image recognition results; L a , L f , L r respectively represent the recognition results obtained by the attack image, the merged image and the restored image; ε γ , ε β , ε ρ respectively represent the proportion of L a , L f , L r . The calculation formula of γ and ρ is as follows: wherein and respectively represent the maximum average value of the specific class label classification parameter in the attack image, the merged image, and the restored image.

Images