Method and device for resisting model inversion attack based on deep learning network structure

Abstract

The invention discloses a method and a device for resisting model inversion attack based on a deep learning network structure. And the risk of model inversion attack is reduced from the data source level by performing block scrambling preprocessing on the image. A deep learning network is designed for an image subjected to block scrambling, and the deep learning network aims to extract symbolic features from the image subjected to block scrambling, improve the recognition accuracy and resist model inversion attacks based on gradient and self-attention. According to the method, features with higher robustness can be extracted from the image subjected to block scrambling, model inversion attacks based on gradient and self-attention are effectively resisted, malicious attackers are prevented from restoring or speculating the original image through technical means such as model inversion, and the method has important privacy protection and information security application value.

Description

Technical Field

[0001] The present invention belongs to the fields of deep learning, computer vision and information security, and specifically relates to a method and device for resisting model inversion attacks based on a deep learning network structure. Background Art

[0002] With the rapid development of deep learning technology, Vision Transformer (ViT) has shown great potential in image processing tasks. Compared with the traditional convolutional neural network (CNN) architecture, ViT shows higher robustness in tasks such as puzzle decoding, object detection, image retrieval, and image segmentation, especially when processing complex data. ViT's Multi-Head Self-Attention (MHSA) mechanism gives it a wider receptive field than CNN, allowing ViT to effectively aggregate information from surrounding areas, thereby better understanding the global features of the image.

[0003] Taking advantage of this feature of ViT, recent studies have explored its application in privacy-preserving image feature extraction and proposed a privacy protection method based on block scrambling. Block scrambling significantly destroys the spatial structure and visual coherence of the original image by randomly disrupting the order of image data blocks, achieves visual de-identification, and effectively protects user privacy. While retaining local data correlation, it breaks the global consistency and does not affect local data integrity, but significantly increases the randomness and unpredictability of the data, thereby effectively enhancing the defense against recovery attacks. For users, block scrambling can effectively prevent the leakage of sensitive information. Even if attackers obtain data through gradient leakage or other methods, they still need to restore the disrupted data structure before extracting meaningful information, which increases the difficulty of information acquisition (such as Figure 1 As shown in Figure 1). According to computational theory, restoring a scrambled sequence is considered an NP problem, because the attacker needs to try a large number of combinations to accurately restore the original data sequence, which requires a lot of computing resources. Therefore, block scrambling has a significant advantage in resisting inversion attacks.

[0004] However, although block scrambling can achieve visual de-identification, it also brings some new challenges when applied to deep neural networks. Existing neural networks either show poor performance or cannot effectively resist attacks when processing block scrambled images. In scenarios with high privacy requirements (such as federated learning), although block scrambling can effectively prevent direct reconstruction of image sequences and thus protect the security of user data, techniques such as DLGA[9] and APRILA

[10] may still allow attackers to infer partial information of the image through statistical analysis, thereby obtaining certain private data (such as Figure 2 Therefore, it is necessary to design a new deep learning network that can maintain high performance under the condition of block scrambling and enhance the resistance to inversion attacks, thereby improving the security and applicability of block scrambling technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to extract iconic features from visually de-identified images that have been subjected to block scrambling processing by constructing a deep learning network, thereby achieving the purpose of effectively performing image analysis and recognition tasks while protecting image privacy.

[0006] A method to resist model inversion attacks based on deep learning network structure.

[0007] Extract features from the input image through a deep learning network;

[0008] The implementation steps include three stages: data preprocessing, deep learning network training, and deep learning network inference:

[0009] (1) In the data preprocessing stage, the original image is scrambled in blocks, and the generated scrambled image is used as the input of the deep learning network;

[0010] (2) In the training phase, the preprocessed images are used to train the deep learning network structure to obtain optimized network parameters;

[0011] (3) In the inference phase, the preprocessed image is input into the deep learning network using the trained network parameters to extract feature identifiers of the scrambled image; based on the aforementioned feature identifiers, the visual task is performed.

[0012] The deep learning network structure includes a block-level Inception aggregation module (PWIA), a serial encoder module, and a query-key skip connection mechanism (QKSC) module;

[0013] (1) The image block-level Inception aggregation module extracts the landmark features within the block from the block-level image. First, it performs multi-scale feature extraction from the perspective of different block sizes of the image, and uses the spatial attention mechanism to filter features at multiple scales. It uses a multi-layer Inception structure to aggregate features, and can extract fine-grained features from the block-level perspective.

[0014] (2) The encoder module is composed of a self-attention module and a Fourier transform gated linear unit. The self-attention module is used to capture the global dependencies between different image blocks. The Fourier transform gated linear unit extracts frequency domain and spatial features respectively by dividing the input features into two branches, namely, the frequency domain and the spatial domain. The frequency domain and spatial features are selectively fused through the gated linear unit, and then normalized through the linear layer and batch normalization, and finally added to the input residual. This module makes full use of the information in the frequency domain and the spatial domain, enhances the diversity of feature expression and the global and local feature capture capability of the model.

[0015] (3) The query-key skip connection mechanism module is used to transfer self-attention queries between different layers

[0016] The role of (Query) and (Key) information is to achieve cross-layer attention sharing and avoid the loss of attention information during the transmission process.

[0017] The image block-level Inception aggregation module extracts features from input image patches through convolution kernels of various sizes, and selectively filters features of different scales in combination with the spatial attention mechanism; through the feature fusion mechanism, the convolution and attention outputs of each scale are integrated to generate embedded features with rich spatial and information of the present invention;

[0018] Divide the input image of C×H×W into multiple The image blocks are divided into image blocks of height and width H×W (where C represents the channel of the image, H represents the height of the image, W represents the width of the image, and p is the block size of the image). Then, convolution kernels of different sizes are applied to each p×p image block (e.g. ) to extract multi-scale features, ensuring that rich local and global information is extracted from each image block; then, the spatial attention module is used to filter the feature maps of each scale and retain the most representative features in each image block; in order to unify the features of different scales, and The feature map generated by the convolution kernel is downsampled to The feature map generated by the convolution kernel has the same size, and the size of the final feature map is resolution; then, the feature maps of all scales are weighted and integrated through the fusion module to generate a comprehensive feature map containing multi-scale information.

[0019] The Fourier transform gated linear unit is used to improve the deep learning model's ability to capture global and local features and the flexibility of information selection;

[0020] The input features are first processed by the linear layer and then divided into three branches, which enter the frequency domain feature extraction path, the spatial domain feature extraction path and the gated signal generation path respectively. The frequency domain feature extraction path, the spatial domain feature extraction path and the gated signal generation path are marked as the first, second and third branches respectively;

[0021] (1) In the first branch path, the input features are converted to the frequency domain by a fast Fourier convolution (FFT Conv) operation. Subsequently, the features are decomposed into real and imaginary parts using a 2D fast Fourier transform (2D FFT), and a 1×1 deep convolution (DW Conv) is applied only on the real part to capture the transformation information of the global features. The real part after convolution is converted back to the time domain by a 2D inverse Fourier transform (2D IFFT) to generate the feature output of the frequency domain branch.

[0022] (2) In the second branch path, the input features are subjected to a 1×1 convolution operation to extract local spatial features, and the spatial features are processed by an activation function to obtain feature outputs in the spatial domain;

[0023] (3) In the third branch path, the input features pass through independent linear layers to generate gating signals, which are used to control the transmission strength of frequency domain and spatial domain features;

[0024] After the output of the first branch and the second branch path, the frequency domain and spatial domain features are respectively multiplied with the gated signal output by the third branch to generate the gated frequency domain features and spatial domain features; the gated frequency domain features and spatial domain features are fused through an addition operation to obtain a joint feature representation.

[0025] The query-key skip connection mechanism is used to realize cross-layer sharing and fusion of query (Query) and key (Key) features in deep neural networks;

[0026] The query-key skip connection mechanism passes the query and key features generated by the multi-head self-attention (MHSA) module of each layer to the next layer after processing by nonlinear activation function (Tanh) and layer normalization, and fuses them with the query and key features generated by the next layer; in this way, the query-key skip connection mechanism ensures the continuity of query and key features between different layers, alleviates the depth degradation caused by residual connections, and enables more effective retention and sharing of key feature information in the shallow stage of the network.

[0027] A device for resisting model inversion attacks based on a deep learning network structure, comprising: processing an input image based on the deep learning network structure using the above-mentioned method.

[0028] In order to effectively resist the model inversion attack, the proposed network is designed to strengthen the anti-attack ability, and uses block scrambling to destroy the spatial structure of the original image, which increases the difficulty for the attacker to restore the image. The above-mentioned deep learning network structure is recorded as JigFormer network structure in the present invention, which includes a block-level Inception aggregation module (PWIA), an encoder module, and a query-key skip connection mechanism (QKSC).

[0029] The block-level Inception aggregation module includes three branches: The module is composed of three branches. The convolution kernels and strides of different scales are applied to the input scrambled image (whose height and width are recorded as H and W respectively) to extract features, and then the key features are strengthened through the spatial attention mechanism. Finally, the multi-scale features are fused to generate a unified feature representation, which is used as the input of the subsequent encoder;

[0030] The first branch: Use a p×p convolution kernel, set the stride to p, and generate a size of Feature map

[0031] The second branch uses The convolution kernel of is set to (Stride) The generated size is Feature map

[0032] The third branch includes The convolution kernel of is set to (Stride) Spatial attention SA, and a 2×2 convolution kernel, with the stride set to 2, generates a size of Feature map

[0033] In the feature fusion stage, the feature maps generated by the second and third branches are concatenated and input into the fusion convolution layer. The fusion convolution layer uses a 2×2 convolution kernel with a stride of 2 and combines it with the spatial attention mechanism to finally fuse it with the feature map generated by the first branch to form a network of size The feature maps are fused as the block-level feature representation of the image;

[0034] The encoder module includes n (n is a positive integer) Fourier transform gating modules, denoted as (FSG0, FSG1, FSG2, ..., FSG n-1 ), Fourier transform gating module FSG i(0≤i≤n-1) includes normalization operation, self-attention module and Fourier transform gated linear unit; the input encoded feature representation enters the normalization layer (Norm) to standardize the data to ensure the stable distribution of inputs in each layer. The standardized data enters the multi-head self-attention module (MHSA) to capture long-distance dependencies in the data. The output after the attention mechanism is added to the original input to form the first residual connection. The output of the self-attention module passes through the normalization layer again to ensure the stability of the data distribution. The normalized data enters the Fourier transform gated linear unit to further extract features. The output of this module is then added to the input of the previous layer to form the second residual connection; FSG i The output result and the query value and key value output by the multi-head self-attention module are used as FSG i+1 Input;

[0035] In the Fourier transform gated linear unit, the input data is processed by the linear layers Fc1 and Fc2 and then divided into three branches, one of which is processed by the FFT convolution module, one is processed by the conventional convolution module, and the other outputs the gated signal V. In the FFT convolution module, the input data is first processed by the FFT convolution, and the data is converted from the time domain to the frequency domain by 2DFFT (two-dimensional fast Fourier transform), and the frequency domain features are decomposed into real and imaginary parts. The real part is subjected to a deep separable convolution (DW Conv1x1) to extract different frequency information. After processing, the data is converted from the frequency domain back to the time domain using 2D IFFT (two-dimensional inverse Fourier transform) for subsequent processing. In the conventional convolution module, the input data passes directly through the conventional convolution layer to extract the time domain features. After the FFT and ordinary convolution modules, the two branches pass through the activation function respectively, and then perform point multiplication with the gated signal to obtain two features X F and X F ,These two feature branches are added to obtain the comprehensive features.

[0036] In the query-key skip connection mechanism, between the i-th layer and the i+1-th layer of the encoder, the QKSC mechanism converts the Query key output of the multi-head attention in the i-th layer into i and Key i After activation and normalization, it is passed to the i+1 layer, thus forming a cross-layer jump connection. This connection allows the information of the previous layer to directly participate in the attention calculation of the subsequent layer, allowing the model to share key information between multiple layers. The specific calculation process is shown in the following formula:

[0037] Q i ,K i ,V i =Linear(Z i )

[0038]

[0039] Q i+1 ,K i+1 ,V i+1 =Linear(Z i+1 )

[0040] Q i+1 =Q i+1 +norm(Tanh(Q i ))

[0041] K i+1 =K i+1 +norm(Tanh(K i ))

[0042]

[0043] in:

[0044] Z i : represents the input of multi-head self-attention at the i-th layer.

[0045] Q i : represents the multi-head self-attention query value at layer i, which is obtained from the input feature Z i Obtained after linear transformation.

[0046] K i : represents the key value at the i-th layer, which is also derived from the input feature Z i After linear transformation, V i : represents the value at layer i, from the input feature Z i Obtained after linear transformation.

[0047] Z i ′ : represents the output features after the i-th layer multi-head self-attention (MHSA) module.

[0048] C: represents the scaling factor, usually the dimension of query and key, used to prevent the attention score from being too large.

[0049] Q i+1 : represents the query value at the i+1 layer, including the query value from the previous layer Q i The accumulated part after Tanh activation and normalization.

[0050] K i+1 :Represents the key matrix at the i+1 layer, including the K i The accumulated part after Tanh activation and normalization.

[0051] Z i ′+1 Represents the output features after the i+1th layer of multi-head self-attention module.

[0052] Linear is a linear transformation function, which is implemented by the fully connected layer in the deep learning model in the present invention. Linear(x)=Wx+b, where W is the weight matrix, b is the bias vector, and x is the input.

[0053] Norm represents the normalization operation, and the LayerNorm implementation used in the present invention can be expressed as μ is the sum of the mean values ​​of the current input x on the d-dimensional features, σ 2 is the variance of the current input x, γ and β are learnable scaling and offset parameters, and ∈ is a non-zero small value to prevent division by zero. The Softmax(x) function converts the input x into a probability distribution. Represents the number of values ​​contained in the input x. i represents the output value corresponding to the i-th element in the input vector currently calculating softmax, which is the target element index of the softmax function. j represents the index of all elements in the input vector.

[0054] Beneficial effects:

[0055] This technology reduces the risk of model inversion attacks from the data source level by preprocessing the original image with block scrambling. In order to adapt to the scrambled image features, a special deep learning network is designed, including the image block-level Inception aggregation module (PWIA), the encoder module (combined with Fourier transform gated linear unit and self-attention mechanism) and the query-key skip connection mechanism (QKSC). This network can extract more robust features from the block-scrambled images, improve the recognition accuracy, and effectively resist the model inversion attacks based on gradient and self-attention, preventing malicious attackers from restoring or inferring the original image, which has important application value in privacy protection and information security.

[0056] Specifically, the present invention discloses a method and device for resisting model inversion attacks based on a deep learning network structure. By preprocessing the image by block scrambling, the risk of model inversion attacks is reduced from the data source level. A deep learning network is designed for block-scrambled images, which aims to extract iconic features from block-scrambled images, improve recognition accuracy, and resist model inversion attacks based on gradients and self-attention. The deep learning network includes three main parts. First, an image block-level Inception aggregation module (PWIA) is adopted to enable the network to associate multi-scale features within the image block and maximize the extraction of unique features of each image block. Then, the Fourier transform gated linear unit (FSGLU) is combined with self-attention as the encoder of the network to optimize the interaction and filtering process of spatial domain and frequency domain information. Finally, the query-key skip connection mechanism (QKSC) is used to propagate the shallow self-attention to the deep layer, enhance the ability of the shallow self-attention to capture global relationships, and achieve a longer average attention range. This invention can extract more robust features from block-scrambled images, and effectively resist model inversion attacks based on gradient and self-attention, preventing malicious attackers from restoring or inferring the original image through technical means such as model inversion. It has important application value in privacy protection and information security.

[0057] The present invention realizes a method and device for resisting model inversion attack based on a deep learning network structure. Compared with the prior art, the present invention has the following advantages:

[0058] 1. Enhanced adaptability: The present invention integrates Fourier shift gated linear units and query-key skip connections to improve the adaptability of deep learning networks to visual de-identification transformations achieved by block scrambling. This adaptability enables it to better handle image block and scrambled feature integration, thereby achieving higher accuracy in visual de-identification image recognition tasks.

[0059] 2. Efficient long-range dependency modeling: By improving the self-attention mechanism, the present invention achieves stronger long-range modeling capabilities in Vision Transformer (algorithm already implemented), effectively solving the problem of Vision Transformer lacking global receptive field at the shallow level. The query-key skip connection mechanism designed by it can transfer information of shallow features to deep layers, allowing the model to optimize and utilize these features at a deeper level, thereby improving recognition accuracy.

[0060] 3. The features extracted from the visually de-identified images achieved by block scrambling are more robust: Tests on the CIFAR-10 and CIFAR-100 datasets show that compared with the existing eleven deep learning network methods, the present invention has achieved the optimal level in the accuracy of visually de-identified image recognition, demonstrating a significant performance advantage in the visually de-identified image recognition task.

[0061] 4. Privacy protection mechanism: The structure designed by the present invention can effectively resist classic gradient attacks and attacks on Vision Transformer (ViT) position embedding, enhance the security of user privacy data, solve the shortcomings of existing technologies in privacy protection, and ensure the safe use of data in an open environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is an application scenario of de-identification in the present invention;

[0063] Figure 2 This is an application scenario of the present invention under high-level privacy requirements;

[0064] Figure 3 It is a schematic diagram of the architecture of the present invention;

[0065] Figure 4 Schematic diagram of the block-level Inception aggregation module described in the present invention (p=16);

[0066] Figure 5 Schematic diagram of the Fourier transform gated linear unit described in the present invention;

[0067] Figure 6 Schematic diagram of the query-key skip connection mechanism described in the present invention;

[0068] Figure 7 Schematic diagram of the original image and the de-identified image processed by the present invention ( Figure 7 (a)-(d) are the original Figure 1 , scrambled image 1, original Figure 2 , scrambled image 2);

[0069] Figure 8 A schematic diagram showing the comparison of the recognition accuracy of scrambled images of the CIFAR-10 dataset using various methods;

[0070] Fig. 9 A schematic diagram showing the comparison of the recognition accuracy of scrambled images of the CIFAR-100 dataset using various methods;

[0071] Fig.10 Schematic diagram of a failed model inversion attack on the present invention ( Fig.10(a)-(d) are respectively: gradient leakage attack input, gradient leakage attack result, learnable position encoding attack input, and learnable position encoding attack result). DETAILED DESCRIPTION

[0072] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0073] In response to the threat of model inversion attacks to image privacy and security, the present invention proposes a method and device for resisting model inversion attacks based on a deep learning network structure, which solves the problem of effectively extracting image features and maintaining high recognition accuracy under block scrambling conditions, and at the same time enhances the ability to resist model inversion attacks based on gradients and self-attention. The deep learning network proposed in the present invention extracts Robin's feature identifiers from the disturbed image as the representation of the final image features, and performs visual tasks such as image classification on the extracted features; at the same time, the network can be used in scenarios such as federated learning for joint training of private data, and can be protected from inversion attacks on the model in federated learning, such as gradient leakage attacks and learnable position encoding attacks, to protect private data.

[0074] Example

[0075] This embodiment provides a method and device for resisting model inversion attacks based on a deep learning network structure, which aims to extract more robust features from block-scrambled images to achieve visual tasks such as image classification, while effectively resisting model inversion attacks based on gradient and self-attention to protect the privacy data security of users.

[0076] For details, see Figure 3 The overall network framework diagram of JigFormer is a sequence-based pattern-building network structure that can be used for both image encoding and image decoding. This embodiment provides a method for image classification. A detailed description of the service, which includes the training phase, the inference phase, and the inversion attack against the model;

[0077] 1. Dataset preparation

[0078] In this embodiment, the CIFAR-10 and CIFAR-100 datasets are used to verify the effectiveness of the invention, where the CIFAR-10 and CIFAR-100 datasets contain 10 and 100 categories of images, respectively, and the size of each image is 32×32 pixels. To meet the input requirements of the model, the images in all datasets are adjusted to a size of 224×224 pixels and randomly shuffled as input to enhance the privacy protection effect. CIFAR-10 and CIFAR-100 contain 50,000 training images and 10,000 test images, respectively, and each category is evenly distributed in the training set and the test set.

[0079] 2. Data preprocessing

[0080] Before inputting the image data into the JigFormer network structure (deep learning network structure), the image data is pre-processed using Algorithm 1. The processed image is as follows Figure 7 As shown:

[0081]

[0082] 3. Model training parameter setting

[0083] The model training uses the AdamW optimizer, which combines the Adam optimization algorithm and weight decay technology to effectively prevent overfitting and improve the generalization ability of the model. The initial learning rate is set to 5e-4, and the cosine decay scheduler is used to gradually reduce the learning rate to achieve more stable training. 128 images are used to update the model parameters in each training iteration, and the model is trained for 300 rounds on all data sets. This embodiment runs in the hardware environment of Nvidia GeForce RTX 3090, implements the solution based on the pytorch deep learning framework, and provides experimental comparisons;

[0084] 4. JigFormer deep learning network structure construction

[0085] Block-level Inception aggregation module, Figure 4 As shown in the figure, the module consists of three branches, which are: The input scrambled image (whose height and width are recorded as H=224 and W=224 respectively) is subjected to convolution kernels of different scales and strides for feature extraction, and then the key features are enhanced through the spatial attention mechanism. Finally, the multi-scale features are fused to generate a unified feature representation, which is used as the input of the subsequent encoder;

[0086] The first branch: Use a p×p convolution kernel, set the stride to p, and generate a size of Feature map

[0087] The second branch uses The convolution kernel of is set to (Stride) The generated size is Feature map

[0088] The third branch includes The convolution kernel of is set to (Stride) Spatial attention SA, and a 2×2 convolution kernel, with the stride set to 2, generates a size of Feature map

[0089] In the feature fusion stage, the feature maps generated by the second and third branches are concatenated and input into the fusion convolution layer. The fusion convolution layer uses a 2×2 convolution kernel with a stride of 2 and combines it with the spatial attention mechanism to finally fuse it with the feature map generated by the first branch to form a network of size The feature maps are fused as the block-level feature representation of the image. In this embodiment, p=16;

[0090] In this embodiment, the encoder module includes 12 Fourier transform gating modules, denoted as (FSG0, FSG1, FSG2, ..., FSG 11 ), Fourier transform gating module FSG i (0≤i≤11) includes normalization operation, self-attention module and Fourier transform gated linear unit; the input encoded feature representation enters the normalization layer (Norm) to standardize the data to ensure the stable distribution of inputs in each layer. The standardized data enters the multi-head self-attention module (MHSA) to capture long-distance dependencies in the data. The output after the attention mechanism is added to the original input to form the first residual connection. The output of the self-attention module passes through the normalization layer again to ensure the stability of the data distribution. The normalized data enters the Fourier transform gated linear unit to further extract features. The output of this module is then added to the input of the previous layer to form the second residual connection; FSG i The output result and the query value and key value output by the multi-head self-attention module are used as FSG i+1 Input;

[0091] In the Fourier transform gated linear unit, Figure 5As shown in the figure, the input data is processed by the linear layers Fc1 and Fc2 and then divided into three branches, one of which is processed by the FFT convolution module, one is processed by the conventional convolution module, and the other outputs the gated signal V. In the FFT convolution module, the input data is first processed by the FFT convolution, and the data is converted from the time domain to the frequency domain by 2D FFT (two-dimensional fast Fourier transform), and the frequency domain features are decomposed into real and imaginary parts. The real part is subjected to a deep separable convolution (DW Conv 1x1) to extract different frequency information. After processing, the data is converted from the frequency domain back to the time domain using 2D IFFT (two-dimensional inverse Fourier transform) for subsequent processing. In the conventional convolution module, the input data directly passes through the conventional convolution layer to extract the time domain features. After the FFT and ordinary convolution modules, the two branches pass through the activation function respectively, and then perform a dot multiplication with the gated signal to obtain two features X F and X F ,These two feature branches are added to obtain the comprehensive features.

[0092] In the query-key skip connection mechanism, Figure 6 As shown, between the i layer and the i+1 layer of the encoder, the QKSC mechanism converts the Query i and Key i After activation and normalization, it is passed to the i+1 layer, thus forming a cross-layer jump connection. This connection allows the information of the previous layer to directly participate in the attention calculation of the subsequent layer, allowing the model to share key information between multiple layers. The specific calculation process is shown in the following formula:

[0093] Q i ,K i ,V i =Linear(Z i )

[0094]

[0095] Q i+1 ,K i+1 ,V i+1 =Linear(Z i+1 )

[0096] Q i+1 =Q i+1 +norm(Tanh(Q i ))

[0097] K i+1 =K i+1 +norm(Tanh(K i ))

[0098]

[0099] Z i : represents the input of multi-head self-attention at the i-th layer.

[0100] Q i : represents the multi-head self-attention query value at layer i, which is obtained from the input feature Z i Obtained after linear transformation.

[0101] K i : represents the key value at the i-th layer, which is also derived from the input feature Z i Obtained after linear transformation.

[0102] V i : represents the value at layer i, from the input feature Z i Obtained after linear transformation.

[0103] Z i ′ : represents the output features after the i-th layer multi-head self-attention (MHSA) module.

[0104] C: represents the scaling factor, usually the dimension of query and key, used to prevent the attention score from being too large.

[0105] Q i+1 : represents the query value at the i+1 layer, including the query value from the previous layer Q i The accumulated part after Tanh activation and normalization.

[0106] K i+1 :Represents the key matrix at the i+1 layer, including the K i The accumulated part after Tanh activation and normalization.

[0107] Z i ′ +1 Represents the output features after the i+1th layer of multi-head self-attention module.

[0108] 5. Comparison of CIFAR-10 and CIFAR-100 image classification experimental results

[0109] Figure 8 and Fig. 9 They respectively represent the final test result outputs on the CIFAR-10 and CIFAR-100 validation sets, and in this embodiment, they are compared with the convolutional neural network, the Transformer architecture neural network, and the MLP architecture neural network.

[0110] The CNN architecture is based on local convolution and is good at capturing local features in undisturbed image structures. However, since the convolution layer mainly relies on the spatial position information of the image, the feature extraction effect of the model is greatly reduced after the image structure is scrambled, resulting in a decrease in accuracy. The self-attention mechanism of the Transformer architecture does not rely on a fixed spatial position, but is based on global feature extraction. Therefore, although the image is scrambled, the self-attention mechanism can still effectively establish the relationship between long-distance features, so that the model still has strong recognition ability when the image structure is disturbed. The layer-by-layer mapping characteristics of the MLP architecture model make it independent of the spatial consistency of the image, and it is relatively robust when processing scrambled images. Since the MLP architecture can aggregate features layer by layer, it has a certain degree of anti-interference in scrambled image tasks, but it is not as good as the Transformer architecture overall.

[0111] JigFormer performs best among all models, which shows that JigFormer has significant anti-interference ability in scrambled image recognition tasks. JigFormer can better cope with image structure scrambling through enhanced equivariant properties and multi-scale feature extraction mechanism. The model is particularly good at balancing the extraction of local and global features, which enables it to maintain excellent recognition performance in scrambled images.

[0112] 6. Model inversion attack on JigFormer deep learning network

[0113] This experiment aims at the privacy protection ability of deep learning models. By performing gradient leakage attacks (existing implemented algorithms) and attention privacy leakage attacks (existing implemented algorithms) on JigFormer and VisionTransformer (ViT), the superiority of JigFormer in privacy protection is verified. The experiment is conducted on CIFAR-10 and CIFAR-100 datasets to demonstrate the effects of different privacy protection strategies. Fig.10 The image reconstructed by the attack is shown in Figure 2.

[0114]

[0115] Table 1: Experimental results of gradient leakage attack on CIFAR-10 dataset

[0116]

[0117] Table 2: Experimental results of attention privacy leakage attack on CIFAR-100 dataset

[0118] Whether it is a gradient leakage attack or an attention privacy leakage attack, JigFormer performs well in indicators such as PSNR, SSIM, and LPIPS, proving that it has significant advantages in privacy protection. This shows that when attacked, the reconstructed images generated by JigFormer are of low quality, poor structural similarity, greater perceptual differences, and difficult to reconstruct. Under both attacks, JigFormer's accuracy is better than ViT, indicating that it can maintain high model performance while strengthening privacy protection. Overall, JigFormer performs significantly better than ViT in privacy protection experiments. Its lower PSNR and SSIM values ​​and higher LPIPS values ​​indicate that it is difficult to restore the original image under reconstruction attacks, effectively ensuring the privacy of the data. In practical applications, JigFormer has stronger anti-attack capabilities and is suitable for scenarios with high requirements for privacy protection.

[0119] In summary, the present invention proposes a method and device for resisting model inversion attacks based on a deep learning network structure. Through unique architecture design and privacy protection mechanism, JigFormer has significant advantages in image classification and privacy protection tasks. It shows high resistance to gradient leakage attacks and attention privacy leakage attacks, making it highly applicable in privacy-sensitive scenarios such as federated learning. At the same time, JigFormer performs well in processing block-scrambled image recognition tasks, can effectively cope with the disturbance of image data, and maintain efficient feature extraction capabilities.

[0120] Therefore, as a method and device for resisting model inversion attacks based on a deep learning network structure, the present invention not only improves the performance of the model in complex visual tasks, but also significantly enhances the ability to resist model inversion attacks. It effectively prevents malicious attackers from restoring or inferring the original image through technical means such as model inversion, and has important privacy protection and information security application value. The invention is suitable for application scenarios that require high privacy protection and high-performance image analysis, and provides a new solution for the security and reliability of deep learning models.

[0121] In addition, the method of the present invention is not limited to being executed in the time sequence described in the specification, and may also be executed in other time sequences, in parallel or independently. Therefore, the execution order of the method described in this specification does not limit the technical scope of the present invention.

[0122] Although the present invention has been disclosed above by describing specific embodiments of the present invention, it should be understood that all the above embodiments and examples are exemplary rather than restrictive. Those skilled in the art may design various modifications, improvements or equivalents of the present invention within the spirit and scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included in the protection scope of the present invention.

Claims

1. A method for resisting model inversion attacks based on a deep learning network structure, characterized in that: Extract features from the input image through a deep learning network; The implementation steps include three stages: data preprocessing, deep learning network training, and deep learning network inference: (1) In the data preprocessing stage, the original image is scrambled in blocks, and the generated scrambled image is used as the depth Learning the input of the network; (2) In the training phase, the preprocessed images are used to train the deep learning network structure to obtain optimized network parameters; (3) In the inference phase, the preprocessed image is input into the deep learning network using the trained network parameters to extract the feature identifiers of the scrambled image; Based on the aforementioned feature identifiers, a visual task is performed.

2. The method for resisting model inversion attack based on deep learning network structure according to claim 1 is characterized in that: The deep learning network structure includes a block-level Inception aggregation module (PWIA), a serial encoder module, and a query-key skip connection mechanism (QKSC) module; (1) The image block-level Inception aggregation module extracts the landmark features within the block from the block-level image. First, it performs multi-scale feature extraction from the perspective of different block sizes of the image and uses the spatial attention mechanism to filter features at multiple scales. Using a multi-layer Inception structure to aggregate features, fine-grained features can be extracted from a block-level perspective; (2) The encoder module is composed of a self-attention module and a Fourier transform gated linear unit. The self-attention module is used to capture the global dependencies between different image blocks. The Fourier transform gated linear unit extracts frequency domain and spatial features by dividing the input features into two branches, namely, the frequency domain and the spatial domain, respectively, and selectively fuses them through a gated linear unit, and then normalizes them through a linear layer and batch normalization, and finally adds them to the input residual. This module makes full use of information in the frequency domain and the spatial domain, enhancing the diversity of feature expression and the model's ability to capture global and local features; (3) The query-key skip connection mechanism module plays the role of transmitting self-attention query (Query) and key (Key) information between different layers, realizing cross-layer attention sharing and avoiding the loss of attention information during the transmission process.

3. The method for resisting model inversion attack based on deep learning network structure according to claim 1 is characterized in that: The image block-level Inception aggregation module extracts features from input image patches through convolution kernels of various sizes, and combines the spatial attention mechanism to selectively filter features of different scales; Through the feature fusion mechanism, the convolution and attention outputs of each scale are integrated to generate embedded features with rich spatial and context information; Divide the H×W input image into multiple The image blocks are divided into image blocks of height and width H×W (where C represents the channel of the image, H represents the height of the image, W represents the width of the image, and p is the block size of the image). Then, convolution kernels of different sizes (such as ) to extract multi-scale features, ensuring that rich local and global information is extracted from each image block; then, the spatial attention module is used to filter the feature maps of each scale and retain the most representative features in each image block; in order to unify the features of different scales, and The feature map generated by the convolution kernel is downsampled to The feature map generated by the convolution kernel has the same size, and the size of the final feature map is resolution; then, the feature maps of all scales are weighted and integrated through the fusion module to generate a comprehensive feature map containing multi-scale information.

4. The method for resisting model inversion attack based on deep learning network structure according to claim 1, characterized in that: The Fourier transform gated linear unit is used to improve the deep learning model's ability to capture global and local features and the flexibility of information selection; The input features are first processed by the linear layer and then divided into three branches, which enter the frequency domain feature extraction path, the spatial domain feature extraction path and the gated signal generation path respectively. The frequency domain feature extraction path, the spatial domain feature extraction path and the gated signal generation path are marked as the first, second and third branches respectively; (1) In the first branch path, the input features are converted to the frequency domain by a fast Fourier convolution (FFT Conv) operation. Subsequently, the features are decomposed into real and imaginary parts using a 2D fast Fourier transform (2D FFT), and a 1×1 deep convolution (DW Conv) is applied only on the real part to capture the transformation information of the global features. The real part after convolution is converted back to the time domain by a 2D inverse Fourier transform (2D IFFT) to generate the feature output of the frequency domain branch. (2) In the second branch path, the input features are subjected to a 1×1 convolution operation to extract local spatial features, and the spatial features are processed by an activation function to obtain feature outputs in the spatial domain; (3) In the third branch path, the input features pass through independent linear layers to generate gating signals, which are used to control the transmission strength of frequency domain and spatial domain features; After the output of the first branch and the second branch path, the frequency domain and space domain features are respectively multiplied with the gated signal output by the third branch to generate gated frequency domain features and space domain features; The gated frequency domain features and spatial domain features are fused through addition operations to obtain a joint feature representation.

5. The method for resisting model inversion attack based on deep learning network structure according to claim 1 is characterized in that: The query-key skip connection mechanism is used to realize cross-layer sharing and fusion of query (Query) and key (Key) features in deep neural networks; The query-key skip connection mechanism passes the query and key features generated by the multi-head self-attention (MHSA) module of each layer to the next layer after processing by nonlinear activation function (Tanh) and layer normalization, and fuses them with the query and key features generated by the next layer; in this way, the query-key skip connection mechanism ensures the continuity of query and key features between different layers, alleviates the depth degradation caused by residual connections, and enables more effective retention and sharing of key feature information in the shallow stage of the network.

6. A device for resisting model inversion attack based on a deep learning network structure, characterized in that: The method comprises processing an input image based on a deep learning network structure using the method described in any one of claims 1 to 5.

Images