Two-stage image privacy protection method and system based on dynamic mask and generation recovery

Abstract

The application discloses a two-stage image privacy protection method and system based on dynamic masks and generation recovery, and the method comprises the following steps: S1, a data set ImgDataset is established, preprocessed, and the categories of each target are labeled and proportionally divided into a training set D tr , a verification set D v , and a test set D te ; the training set D tr is classified and trained; S2, the key attention area of an image is distinguished, a corresponding binary mask is generated, and the original image is masked by using the mask; S3, the masked image is input into a generator for training, and the trained parameters are encrypted; S4, the masked image and the parameter ciphertext obtained in S3 are respectively sent to a receiving end, the parameter ciphertext is decrypted in the receiving end, the network weight parameters are loaded into the generator with the same network structure in the receiving end as that in the sending end, and the generator repairs the masked image. By encrypting the training model parameters, the application avoids encrypting massive image data, reduces the calculation cost, and improves the encryption and decryption speed.

Description

Technical Field

[0001] This invention belongs to the field of image information privacy protection technology based on deep learning, and specifically relates to an image privacy protection method and system based on two stages of dynamic masking and generation and restoration during image transmission. Background Technology

[0002] With the emergence and widespread use of digital products such as digital cameras and smartphones, digital images are undoubtedly the fastest-growing type of multimedia information, and are widely used in many fields such as engineering, defense, medicine, and scientific experiments. Therefore, image information security has become an important agenda item.

[0003] Existing image encryption methods can be divided into two types: full-image encryption and selective encryption. The former encrypts the entire image and has good encryption effect, but this method has high computational complexity and cannot meet the requirements of rapidly expanding image data; the latter strikes a balance between security and computational complexity by using salient object detection algorithms to identify important regions of the image and encrypt those regions. This type of method can reduce computational costs to some extent.

[0004] While both methods offer some encryption effectiveness, full-image encryption, while ensuring encryption quality, cannot simultaneously improve encryption and decryption speed. Furthermore, due to the scarcity of salient object datasets, salient object detection algorithms cannot be widely applied to various scenarios, thus limiting their practicality for extracting key image regions. In the information and intelligent era, both methods require image encryption. Digital images are large in volume and highly redundant, resulting in high encryption costs. Moreover, the capacity for manual processing of massive amounts of data is limited. Therefore, using intelligent systems to process images is gradually becoming a new trend. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a two-stage image privacy protection method and system based on dynamic masking and generative reconstruction. In the transmission stage, this invention provides an adaptive masking method that dynamically generates a mask, then automatically and adaptively occludes sensitive areas of the image, hiding effective information. Simultaneously, the masked image is input into a generator for training, and the trained parameters are then encrypted. Finally, the masked image and ciphertext are sent separately to the receiving end. In the receiving stage, the ciphertext of the network parameter set is first decrypted, then the decrypted plaintext parameters are loaded into the generator, and generative reconstruction of the masked image is performed. Finally, the image is processed. If the image is illegally intercepted during transmission, the recognition model cannot locate or detect the image. Furthermore, this invention transforms image encryption into encryption of the generator's training parameters, effectively improving encryption / decryption speed and reducing computational costs. Moreover, this invention designs a region-aware adaptive category activation loss function, which can further improve the quality of the generated image.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] A two-stage image privacy protection method based on dynamic masking and generative restoration includes the following steps:

[0008] S1. Create a dataset ImgDataset, preprocess it, assign class labels to each target, and proportionally divide it into training set D. tr Validation set D v Test set D te For training set D tr Conduct categorized training.

[0009] S2. Adaptively identify the key areas of interest in the image, generate corresponding binary masks, and use the masks to mask the original image.

[0010] S3. Input the masked image into the generator for training. Encrypt the trained network weight parameters.

[0011] S4. The mask image and the ciphertext of parameters obtained in step S3 are sent to the receiving end respectively. The receiving end decrypts the ciphertext of parameters and loads the network weight parameters into the generator in the receiving end, which has the same network structure as the sending end. The generator repairs the mask image.

[0012] Preferably, step S1 specifically includes scaling the ImgDataset to a uniform size of 256*256, converting the image data into tensors and normalizing them, then inputting them into ResNet18 for training, and recording D. v Average accuracy and cross-entropy loss are used to save the model parameters that best validate performance.

[0013] Preferably, step S2 includes the following steps:

[0014] S21, D tr The input is fed into a trained ResNet18 network, resulting in a one-dimensional vector of length C. The softmax classifier is then used to obtain the prediction result y and its corresponding class label c.

[0015] S22, extract the feature map F after the last convolutional layer of ResNet18, which contains 512 channels (f i Let f be the i-th channel of F, and each channel has a size of 8*8. Then, f... i Perform bilinear interpolation to enlarge to 256*256 to obtain f i ′, f i After multiplying by the original image, the result is fed back into the trained ResNet18, and the output is passed through a softmax classifier to extract the prediction result y corresponding to the class label c. i As shown in formula (1), using y i The difference between y and y is used to evaluate the contribution of the i-th channel in feature map F to category c, and the negative value is reduced to zero by a ReLU function to obtain the score class activation map (Score-CAM, ScoreClassActivation Mapping) CAM0; as shown in formula (2), CAM0 is standardized by deviation to obtain the class activation map CAM.

[0016]

[0017]

[0018] S23, set a threshold γ (0.5 < γ < 1), and pixels in the category activation map CAM with a value greater than or equal to γ ​​can be regarded as the key areas of the original image.

[0019] S24. Construct a 256*256 all-zero mask. Then search for the pixel with the largest pixel value in the category activation image that is greater than γ, obtain its position coordinates, set the pixel value of the pixel to 0, and set the pixel in the same position in the mask to 1.

[0020] S25. Repeat step S24 until all pixel values ​​in CAM are less than γ. A binary mask is obtained, whose pixel values ​​are either 0 or 1. 1 corresponds to important pixels in the original image, and 0 corresponds to others.

[0021] S26. Fill the pixels in the original image that correspond to the positions with a pixel value of 1 in the binary mask with an integer from 0 to 255, and finally obtain the corresponding mask image.

[0022] Preferably, step S3 includes the following steps:

[0023] S31, Select the mask image I from S2 mask and its original image I gt The input is fed into the generator, where the encoder identifies the masked regions of the image and extracts its semantic information.

[0024] S32, the decoder in the generator upsamples the extracted semantic information to generate a preliminary repair result image.

[0025] S33, the discriminator receives the preliminary result image I output by the generator. re The adversarial loss is calculated based on the original image and returned to the generator.

[0026] S34, the parameters of the trained generator are encrypted based on the Pell sequence and elliptic curves.

[0027] Specifically, in S31, the encoder employs an eight-group structure. The first group consists of a single convolutional layer; the second to seventh groups each contain an activation layer, a convolutional layer, and a batch normalization layer; and the eighth group contains an activation layer and a single convolutional layer. The number of convolutional kernels in the first, second, and third groups increases from 3 to 64, 64 to 128, and 128 to 256, respectively. The number of convolutional kernels in the fourth to eighth groups is 512. All eight groups have a convolutional size of 4, a stride of 2, and padding of 1. The activation function for the activation layers in the second to eighth groups is LeakyReLU. The encoder ultimately outputs 512 feature maps of size 1*1.

[0028] Specifically, in S32, the decoder performs deconvolution operations on the extracted semantic information features, employing a nine-group structure. Groups one through seven consist of one activation layer, one deconvolution layer, and one batch normalization layer; group eight consists of one activation layer and one deconvolution layer; and group nine has only one activation layer. The semantic information features output from the encoder's first through eighth groups (from front to back) are stacked with the semantic information features output from the decoder's eighth through eighth groups (from back to front), ensuring that each layer of the decoder can acquire both the semantic information features output from the previous layer and those output from the encoder. The first through fourth groups have 512 convolution kernels, while the fifth through eighth groups have kernel numbers decreasing from 1024 to 256, 512 to 128, 256 to 64, and 128 to 3, respectively. The deconvolution kernels in groups one through eight all have a size of 4, a stride of 2, padding of 1, and the activation function is ReLU. The activation function in group nine is Tanh. The decoder eventually outputs a preliminary result image of 256*256 pixels with 3 channels.

[0029] Specifically, in S33, the Markov discriminator mainly consists of four convolutional layers, three LeakyReLU activation functions, and one Sigmoid activation function layer. The first three convolutional layers have a kernel size of 4, a stride of 2, and a padding size of 1. The LeakyReLU activation function is used to activate neurons, with a negative slope of 0.2, and the implace parameter is set to perform the operation in situ. The last convolutional layer has a kernel size of 4, a stride of 1, and a padding size of 1, and uses the Sigmoid activation function to activate neurons. Local discrimination is performed using the preliminary result image, ensuring the effective recovery of low-frequency local texture structures. The Markov discriminator receives the preliminary result image and, based on the preliminary result image and its original complete image, performs discriminative processing to obtain the adversarial loss L. adv :

[0030]

[0031] In formula (3), G represents the generator, D represents the Markov discriminator, E(*) represents the expected value of the distribution function, and p data (I gt ) represents the distribution of the true sample, p data (I mask D(I) represents the distribution of the mask image. gt ) indicates that the Markov discriminator predicts I. gt The probability, D(I) mask ) indicates that the Markov discriminator predicts I. mask The probability of.

[0032] While the preliminary result image and its original complete image are input into the discriminator, the pixel-level loss L between the preliminary result image and its original complete image is also calculated. re :

[0033] L re =||I re -I gt ||1(4)

[0034] The preliminary result image and its original complete image are input into the VGG16 feature extractor, which calculates the perceptual loss L at the feature level between the images. p With style loss L s :

[0035]

[0036]

[0037]

[0038] In formula (5), N represents the number of convolutional layers in VGG16, so N = 5 here; Φ j C represents the j-th convolutional layer in VGG16; j H j W j represents the size of the output feature map of the j-th convolutional layer. C, H, and W represent the number of channels, height, and width of the output feature map of the convolutional layer, respectively. The Gram matrix operation in formula (6) is as shown in formula (7), where m,n represent the index of the input feature map F in the channel dimension, and p represents the index of the feature map F in the spatial dimension.

[0039] In addition to the aforementioned losses, to improve the quality of the generated image, this invention designs a region-aware category activation loss function L. CAM :

[0040]

[0041] In formula (8) Indicate I gt Category activation graph, Indicate I re The category activation map, where CHW represents the number of pixels in the category activation map.

[0042] Pixel-level loss, perceptual loss, style loss, class activation loss, and adversarial loss from the discriminator output are returned to the training generator. The weights of each loss function are 1, 0.01, 0.1, and 0.2%, respectively. 0.002, n represents I mask The number of pixels in the mask region is used because This is because CAM focuses on key areas in an image. If an adaptive mask is applied to these areas, the number of mask pixels will vary in each image. Therefore, using dynamic weighted parameters is more beneficial for parameter optimization.

[0043] Specifically, the encryption process in S34 is as follows: First, plaintext is diffused by using cyclic shift on the symbol set to obtain meaningless plaintext. Second, each element of the diffused text is encoded as a real number using the Pell sequence and binary sequence to hide the diffused text elements. Finally, the encoded diffused text is obfuscated by generating permutations on elliptic curves.

[0044] Preferably, step S4 includes the following steps:

[0045] S41, the sending end sends the parameter ciphertext and the mask image to the receiving end respectively.

[0046] S42, the receiving end receives the mask image and encrypted parameter ciphertext from the sending end, and decrypts the ciphertext to obtain the parameter plaintext;

[0047] S43, load the parameters into the generator, the generator generates and restores the image, and then inputs the restored image into the depth recognition model, the model recognizes and locates the image.

[0048] This invention also discloses a two-stage image privacy protection system based on dynamic masking and generative restoration, which is used to perform the above method and includes the following modules:

[0049] Dataset creation module: Creates the dataset ImgDataset, preprocesses it, assigns class labels to each target, and divides it into training set D in a proportional manner. tr Validation set D v Test set D te For training set D tr Perform classification training;

[0050] Masking module: Adaptively identifies key regions of interest in the image, generates corresponding binary masks, and uses the masks to mask the original image;

[0051] Encryption module: Inputs the mask image into the generator for training and encrypts the trained network weight parameters;

[0052] Repair module: The mask image and parameter ciphertext are sent to the receiving end respectively. The receiving end decrypts the parameter ciphertext and loads the parameters into a generator with the same network structure as the sending end. The generator repairs the mask image.

[0053] Compared with the prior art, the beneficial effects of this invention are:

[0054] (1) This invention avoids encrypting massive image data by encrypting the training model parameters, reduces computational costs, increases encryption and decryption speed, and the entire framework can run in an end-to-end manner.

[0055] (2) This invention provides an adaptive masking method based on category activation maps, which can automatically and adaptively mask important information in images. It also proposes a category activation loss function based on region perception and adaptation, which can improve the quality of the restored image and make it easier for intelligent recognition devices to identify and locate.

[0056] (3) The technical solution proposed in this invention has high security. Because existing generators perform poorly in repairing untrained images, even if the masked image is intercepted, the interceptor cannot repair it using the repair model. Furthermore, if the key is leaked, the interceptor is unaware of the network structure and cannot repair the image. Only when the masked image, network parameters, and key are all intercepted can the interceptor obtain the image information. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the architecture of a two-stage image privacy protection method based on dynamic masking and generation restoration according to a preferred embodiment of the present invention;

[0058] Figure 2 These are example images of image inpainting trained by the generator;

[0059] Figure 3 This is an example image of image inpainting where the generator was not trained.

[0060] Figure 4 This is a block diagram of a two-stage image privacy protection system based on dynamic masking and generation restoration, according to a preferred embodiment of the present invention. Detailed Implementation

[0061] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] like Figure 1-3 As shown, this invention discloses a two-stage image privacy protection method based on dynamic masking and generative restoration, comprising the following steps:

[0063] S1. Create a dataset ImgDataset, preprocess it, label the categories of each object, and divide it into training set D in a 7:2:1 ratio. tr Validation set D v Test set D te Then ResNet18 was used to process D. tr Conduct categorized training.

[0064] S2. Use Score-CAM (Class Activation Mapping) to adaptively identify the key areas of interest in the image, generate the corresponding binary mask using pixel replacement, and then use the mask to mask the original image.

[0065] S3. Feed the masked image into the generator for training. Encrypt the trained parameters based on the Pell sequence and elliptic curves.

[0066] S4. The mask image and the ciphertext of parameters obtained in step S3 are sent to the receiving end respectively. The receiving end decrypts the ciphertext of parameters and loads the network weight parameters into the generator in the receiving end, which has the same network structure as the sending end. The generator repairs the mask image.

[0067] The steps of this embodiment will be described in detail below.

[0068] In step S1 of this embodiment, the ImgDataset is uniformly scaled to a size of 256*256, the image data is converted into tensors and normalized, and then input into ResNet18 for training, recording D.v Average accuracy and cross-entropy loss are used to save the model parameters that best validate performance.

[0069] Step S2 in this embodiment includes the following steps:

[0070] S21, D tr The input is fed into a trained ResNet18 network, resulting in a one-dimensional vector of length C. The softmax classifier is then used to obtain the prediction result y and its corresponding class label c.

[0071] S22, extract the feature map F after the last convolutional layer of ResNet18, which contains 512 channels (f i Let f be the i-th channel of F, and each channel has a size of 8*8. Then, f... i Perform bilinear interpolation to enlarge to 256*256 to obtain f i ′, f i After multiplying by the original image, the result is fed back into the trained ResNet18, and the output is passed through a softmax classifier to extract the prediction result y corresponding to the class label c. i As shown in formula (1), using y i The difference between y and y is used to evaluate the contribution of the i-th channel in the feature map F to the category label c, and the negative value is reduced to zero by a ReLU function to obtain CAM0; as shown in formula (2), CAM0 is standardized by deviation to obtain the category activation map CAM.

[0072]

[0073]

[0074] S23, set a threshold γ (0.5 < γ < 1), and pixels in the category activation map CAM with a value greater than or equal to γ ​​can be regarded as the key areas of the original image.

[0075] S24. Construct a 256*256 all-zero mask. Then search for the pixel with the largest pixel value in the category activation image that is greater than γ, obtain its position coordinates, set the pixel value of the pixel to 0, and set the pixel in the same position in the mask to 1.

[0076] S25. Repeat step S24 until all pixel values ​​in CAM are less than γ. A binary mask is obtained, whose pixel values ​​are either 0 or 1. 1 corresponds to important pixels in the original image, and 0 corresponds to others.

[0077] S26. Fill the pixels in the original image that correspond to the positions with a pixel value of 1 in the binary mask with an integer from 0 to 255, and finally obtain the corresponding mask image.

[0078] Step S3 in this embodiment includes the following steps:

[0079] S31, Select the mask image I from S2 mask and its original image I gt The input is fed into the generator, where the encoder identifies the masked regions of the image and extracts its semantic information.

[0080] S32, the decoder in the generator upsamples the extracted semantic information to generate a preliminary repair result image.

[0081] S33, the discriminator receives the preliminary result image I output by the generator. re The adversarial loss is calculated based on the original image and returned to the generator.

[0082] S34, the parameters of the trained generator are encrypted based on the Pell sequence and elliptic curves.

[0083] More specifically, in step S31, the encoder employs an eight-group structure. The first group consists of a single convolutional layer; the second to seventh groups each contain an activation layer, a convolutional layer, and a batch normalization layer; and the eighth group contains an activation layer and a single convolutional layer. The number of convolutional kernels in the first, second, and third groups is increased from 3 to 64, 64 to 128, and 128 to 256, respectively. The number of convolutional kernels in the fourth to eighth groups is 512. All eight groups have a convolutional size of 4, a stride of 2, and padding of 1. The activation function for the activation layers in the second to eighth groups is LeakyReLU. The encoder ultimately outputs 512 feature maps of size 1*1.

[0084] In step S32, the decoder performs deconvolution operations on the extracted semantic information features, employing a nine-group structure. Groups one through seven consist of one activation layer, one deconvolution layer, and one batch normalization layer; group eight consists of one activation layer and one deconvolution layer; and group nine consists of only one activation layer. The semantic information features output from the encoder's first through eighth groups (from front to back) are stacked with the semantic information features output from the decoder's eighth through eighth groups (from back to front), ensuring that each layer of the decoder can acquire both the semantic information features output from the previous layer and those output from the encoder. The first through fourth groups have 512 convolution kernels, while the fifth through eighth groups have kernel numbers decreasing from 1024 to 256, 512 to 128, 256 to 64, and 128 to 3, respectively. The deconvolution kernels in groups one through eight all have a size of 4, a stride of 2, a padding of 1, and all use ReLU activation functions. The ninth group uses Tanh activation function. The decoder eventually outputs a preliminary result image of 256*256 pixels with 3 channels.

[0085] In step S33, the Markov discriminator mainly consists of four convolutional layers, three LeakyReLU activation functions, and one Sigmoid activation function layer. The first three convolutional layers have a kernel size of 4, a stride of 2, and a padding size of 1. The LeakyReLU activation function is used to activate neurons, with a negative slope of 0.2, and the implace parameter is set to perform the operation in situ. The last convolutional layer has a kernel size of 4, a stride of 1, and a padding size of 1, and the Sigmoid activation function is used to activate neurons. Local discrimination is performed using the preliminary result image, ensuring the effective recovery of low-frequency local texture structures. The Markov discriminator receives the preliminary result image and, based on the preliminary result image and its original complete image, performs discriminative processing to obtain the adversarial loss L. adv :

[0086]

[0087] In formula (3), G represents the generator, D represents the Markov discriminator, E(*) represents the expected value of the distribution function, and p data (I gt ) represents the distribution of the true sample, p data (I mask D(I) represents the distribution of the mask image. gt ) indicates that the Markov discriminator predicts I. gt The probability, D(I) mask ) indicates that the Markov discriminator predicts I. mask The probability of.

[0088] While the preliminary result image and its original complete image are input into the discriminator, the pixel-level loss L between the preliminary result image and its original complete image is also calculated. re :

[0089] L re =||I re -I gt ||1(4)

[0090] The preliminary result image and its original complete image are input into the VGG16 feature extractor, which calculates the perceptual loss L at the feature level between the images. p With style loss L s :

[0091]

[0092]

[0093]

[0094] In formula (5), N represents the number of convolutional layers in VGG16, so N = 5 here; Φ j C represents the j-th convolutional layer in VGG16; j H j W j represents the size of the output feature map of the j-th convolutional layer. C, H, and W represent the number of channels, height, and width of the output feature map of the convolutional layer, respectively. The Gram matrix operation in formula (6) is as shown in formula (7), where m,n represent the index of the input feature map F in the channel dimension, and p represents the index of the feature map F in the spatial dimension.

[0095] In addition to the aforementioned losses, to improve the quality of the generated image, this invention designs a region-aware category activation loss function L. CAM :

[0096]

[0097] In formula (8) Indicate I gt Category activation graph, Indicate I re The category activation map, where CHW represents the number of pixels in the category activation map.

[0098] Pixel-level loss, perceptual loss, style loss, class activation loss, and adversarial loss from the discriminator output are returned to the training generator. The weights of each loss function are 1, 0.01, 0.1, and 0.2%, respectively. 0.002, n represents I mask The number of pixels in the mask region is used because This is because CAM focuses on key areas in an image. If an adaptive mask is applied to these areas, the number of mask pixels will vary in each image. Therefore, using dynamic weighted parameters is more beneficial for parameter optimization.

[0099] The encryption method in step S34 is as follows: First, plaintext is diffused by using cyclic shift on the symbol set to obtain meaningless plaintext. Second, each element of the diffused text is encoded as a real number using the Pell sequence and binary sequence to hide the diffused text elements. Finally, the encoded diffused text is obfuscated by generating permutations on an elliptic curve.

[0100] Step S4 in this embodiment specifically includes the following steps:

[0101] S41, the sending end sends the parameter ciphertext and the mask image to the receiving end respectively.

[0102] S42, the receiving end receives the mask image and encrypted parameter ciphertext from the sending end, and decrypts the ciphertext to obtain the parameter plaintext.

[0103] S43, load the parameters into the generator, the generator generates and restores the image, and then inputs the restored image into the depth recognition model, the model recognizes and locates the image.

[0104] like Figure 1 , 4 As shown, this embodiment discloses a two-stage image privacy protection system based on dynamic masking and generative restoration, which is used to perform the above method and includes the following modules:

[0105] Dataset creation module: Creates the dataset ImgDataset, preprocesses it, assigns class labels to each target, and divides it into training set D in a proportional manner. tr Validation set D v Test set D te For training set D tr Perform classification training;

[0106] Masking module: Adaptively identifies key regions of interest in the image, generates corresponding binary masks, and uses the masks to mask the original image;

[0107] Encryption module: Inputs the mask image into the generator for training and encrypts the trained network weight parameters;

[0108] Repair module: The mask image and parameter ciphertext are sent to the receiving end respectively. The receiving end decrypts the parameter ciphertext and loads the parameters into a generator with the same network structure as the sending end. The generator repairs the mask image.

[0109] Other aspects of this embodiment can be found in the above method embodiments.

[0110] The above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A two-stage image privacy protection method based on dynamic masking and generative restoration, characterized by: Includes the following steps: S1. Create a dataset ImgDataset, preprocess it, assign class labels to each target, and divide it into training set D in proportion. tr Validation set D v Test set D te For training set D tr Perform classification training; S2. Adaptively identify the key areas of interest in the image, generate the corresponding binary mask, and use the mask to mask the original image; S3. Input the masked image into the generator for training, and encrypt the trained network weight parameters. S4. Send the mask image and the parameter ciphertext obtained in step S3 to the receiving end respectively. The receiving end decrypts the parameter ciphertext and loads the network weight parameters into the generator in the receiving end, which has the same network structure as the sending end. The generator repairs the mask image. Step S2 specifically includes the following steps: S21, training set D tr The input is fed into a trained ResNet18 network to obtain a one-dimensional vector of length C. The softmax classifier is then used to obtain the prediction result y and its corresponding class label c. S22, extract the feature map F after the last convolutional layer of ResNet18, which contains 512 channels, f i Let F be the i-th channel, and the size of each channel is... ; will f i Perform bilinear interpolation to amplify get , After multiplying with the original image, the result is fed back into the trained ResNet18, and the output is passed through a softmax classifier to extract the prediction result y corresponding to class c. i As shown in formula (1), y i The difference between y and y is used to evaluate the contribution of the i-th channel in the feature map F to the category label c, and the negative value is reduced to zero by the ReLU function to obtain the score category activation map CAM0; as shown in formula (2), CAM0 is standardized by deviation to obtain the category activation map CAM; (1) (2) S23, set a threshold γ, 0.5 < γ < 1, and the pixel values ​​in the category activation map CAM that are greater than or equal to γ ​​are regarded as the key areas of the original image; S24, construct a The all-zero mask searches for the pixel with the largest pixel value in the category activation image that is greater than γ, obtains its position coordinates, sets the pixel value of the pixel to 0, and sets the pixel of the same position in the mask to 1. S25. Repeat step S24 until all pixel values ​​in CAM are less than γ, and obtain a binary mask whose pixel values ​​are either 0 or 1. 1 corresponds to important pixels in the original image, and 0 corresponds to others. S26, fill the pixels in the original image that correspond to the positions with a pixel value of 1 in the binary mask with an integer from 0 to 255 to obtain the corresponding mask image; Step S3 includes the following steps: S31, Select the mask image from step S2. and its original image The input is fed into the generator, where the encoder identifies the masked regions of the image and extracts its semantic information. S32, the decoder in the generator upsamples the extracted semantic information to generate a preliminary repair result image; S33, the discriminator receives the preliminary result image output by the generator. Calculate the adversarial loss based on the original image and return it to the generator; S34, the parameters of the trained generator are encrypted based on the Pell sequence and elliptic curves.

2. The two-stage image privacy protection method based on dynamic masking and generative restoration as described in claim 1, characterized in that, Step S1 specifically includes scaling the dataset ImgDataset uniformly to [the specified scale]. The image data is converted into tensors and normalized, then input into a ResNet18 for training, and the validation set D is recorded. v Average accuracy and cross-entropy loss are used to save the model parameters that best validate performance.

3. The two-stage image privacy protection method based on dynamic masking and generative restoration as described in claim 1, characterized in that, In step S31, the encoder adopts an eight-group structure. The first group consists of one convolutional layer, the second to seventh groups consist of one activation layer, one convolutional layer, and one batch normalization layer, and the eighth group consists of one activation layer and one convolutional layer. The number of convolutional kernels in the first, second, and third groups is increased from 3 to 64, 64 to 128, and 128 to 256, respectively. The number of convolutional kernels in the fourth to eighth groups is 512. The convolutional size in all eight groups is 4, the stride is 2, and the padding is 1. The activation function of the activation layers in the second to eighth groups is LeakyReLU. The encoder finally outputs 512 images with a size of [missing information]. The feature map.

4. The two-stage image privacy protection method based on dynamic masking and generative restoration as described in claim 3, characterized in that, In step S32, the decoder performs deconvolution operations on the extracted semantic information features, using a nine-group structure. The first to seventh groups consist of one activation layer, one deconvolution layer, and one batch normalization layer; the eighth group consists of one activation layer and one deconvolution layer; and the ninth group consists of only one activation layer. The semantic information features output from the encoder's first to eighth groups from front to back are stacked with the semantic information features output from the decoder's eighth to first groups from back to front. The number of convolution kernels in the first to fourth groups of deconvolution is 512, while the number of convolution kernels in the fifth to eighth groups decreases from 1024 to 256, 512 to 128, 256 to 64, and 128 to 3, respectively. The deconvolution size in the first to eighth groups is 4, the stride is 2, the padding is 1, and the activation function is ReLU. The activation function in the ninth group is Tanh. The decoder ultimately outputs a 3-channel image. Preliminary results diagram.

5. The two-stage image privacy protection method based on dynamic masking and generative restoration as described in claim 4, characterized in that, In step S33, the discriminator mainly consists of four convolutional layers, three LeakyReLU activation functions, and one Sigmoid activation function layer. The first three convolutional layers have a kernel size of 4, a stride of 2, and a padding size of 1. The LeakyReLU activation function is used to activate neurons, with a negative slope of 0.2 and the implace parameter selected for in-situ operation. The last convolutional layer has a kernel size of 4, a stride of 1, and a padding size of 1. The Sigmoid activation function is used to activate neurons. The discriminator receives the preliminary result image and, based on the preliminary result image and its original complete image, performs discriminative processing to obtain the adversarial loss. : (3) In formula (3), Represents a generator. This represents a Markov discriminator. This represents the expected value of the distribution function. Represents the distribution of the real samples. This represents the distribution of the mask image. This indicates that the Markov discriminator predicts... The probability, This indicates that the Markov discriminator predicts... The probability of; While the preliminary result image and its original complete image are input into the discriminator, the pixel-level loss between the preliminary result image and its original complete image is also calculated. : (4) The preliminary result image and its original complete image are input into the VGG16 feature extractor, which calculates the perceptual loss at the feature level between the images. With style loss : (5) (6) (7) In formula (5), This indicates the number of convolutional layers in VGG16, so here... ; Indicates the first in VGG16 One convolutional layer; Indicates the first The size of the output feature map of each convolutional layer; , , These represent the number of channels, height, and width of the output feature map of the convolutional layer, respectively; in formula (6) Matrix operations are performed as shown in formula (7), where, Represents the input feature map Indexing along the channel dimension Representation of feature map Index in spatial dimensions; Design a region-aware category activation loss function. : (8) In formula (8), express Category activation graph, express Category activation graph, This indicates the number of pixels in the category activation map; Pixel-level loss, perceptual loss, style loss, class activation loss, and adversarial loss from the discriminator output are returned to the training generator; the weights of each loss function are 1, 0.01, 0.1, and 0.1, respectively. 0.002 express The number of pixels in the masked region.

6. The two-stage image privacy protection method based on dynamic masking and generative restoration as described in claim 5, characterized in that, The encryption in step S34 is as follows: First, plaintext is diffused by using cyclic shift on the symbol set to obtain meaningless plaintext; second, each element of the diffused text is encoded as a real number using the Pell sequence and binary sequence to hide the diffused text elements; finally, the encoded diffused text is obfuscated by generating permutations on elliptic curves.

7. The two-stage image privacy protection method based on dynamic masking and generative restoration as described in claim 1, characterized in that, Step S4 includes the following steps: S41, the sending end sends the parameter ciphertext and the mask image to the receiving end respectively; S42, the receiving end receives the mask image and encrypted parameter ciphertext from the sending end, and decrypts the ciphertext to obtain the parameter plaintext; S43, load the parameters into the generator, the generator generates and restores the image, and then inputs the restored image into the depth recognition model, the model recognizes and locates the image.

8. A two-stage image privacy protection system based on dynamic masking and generative restoration, for performing the method as described in any one of claims 1-7, characterized in that: Includes the following modules: Dataset creation module: Creates the dataset ImgDataset, preprocesses it, assigns class labels to each target, and divides it into training set D in a proportional manner. tr Validation set D v Test set D te For training set D tr Perform classification training; Masking module: Adaptively identifies key regions of interest in the image, generates corresponding binary masks, and uses the masks to mask the original image; Encryption module: Inputs the mask image into the generator for training and encrypts the trained network weight parameters; Repair module: The mask image and parameter ciphertext are sent to the receiving end respectively. The receiving end decrypts the parameter ciphertext and loads the network weight parameters into the generator in the receiving end, which has the same network structure as the sending end. The generator repairs the mask image.

Images