Lightweight Image Data Privacy Protection Device and Method

Abstract

The present invention provides a lightweight image data privacy protection device and method, belonging to the field of network security technology, including: an image feature encoder module, which is used to efficiently capture feature information from an image and convert it into a series of feature maps; an image anonymization module, which is used to eliminate the privacy information contained in the feature maps and generate anonymized feature maps; an image generation module, which is used to reconstruct the anonymized feature maps and finally generate anonymized images. The present invention reduces the number of parameters, reduces the computational amount of the model, saves the training time, can efficiently and accurately extract image features, and improves the feature extraction ability of the model; using the greedy idea, it realizes the anonymization of image features, and can efficiently reconstruct the image after removing the identity from the average image feature map. The reconstructed image not only realizes the privacy protection of the image, but also retains some image information irrelevant to the identity.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and particularly relates to a lightweight image data privacy protection device and method. Background Art

[0002] Biometric recognition technology has made breakthroughs and is widely used in the field of identity authentication, but it has also increased the risk of privacy leakage, especially face recognition technology. For example, images containing people's identity information may be misused. Some malicious users may mine a large amount of privacy-sensitive specific personal information on these websites and quickly spread it on the network, or conduct human flesh searches based on publicly available data to track others, or some private institutions can even identify specific individuals by combining media data. To eliminate such risks, it is necessary to ensure the convenient application of face recognition technology in various scenarios and also strengthen the research and development of relevant privacy protection technologies for information.

[0003] A common method for image privacy protection is anonymization. The key to this method lies in removing the identifier information in the image data to prevent the abuse of identity information. The most commonly used visual privacy protection methods all modify the sensitive areas in the picture by processing the image or video to hide the privacy information of specific individuals. According to the different ways of processing images, privacy protection can be divided into the following two categories: The first category is the method based on data distortion or suppression. The main idea of this method is to use an image filter to process the sensitive areas in the image, making the sensitive areas in the image distorted or covered to hide or remove sensitive information. The problem with this method is that it destroys other useful information in the image and the degree of anonymization is insufficient. In addition, statistical models or probability models can also be used to adjust the corresponding images, such as the k-anonymity algorithm. This method aggregates the k most similar images and then replaces the original images with the aggregated images, making each original image indistinguishable from the other k-1 images, thus achieving k-anonymity privacy protection; The second category of methods is image encryption: The main idea of this method is to encode the image data to make the original data difficult to understand in order to achieve reliable storage security and content transmission security. The security of this method largely depends on the strength of the key used. Unauthorized users cannot extract the valid information in the encrypted video, and only users with the correct decryption key can extract the valid information.

[0004] The problem of privacy protection of information in images, as a research problem with important application value, has attracted extensive attention in the academic and industrial circles in terms of privacy protection. In recent years, due to the emergence of deep learning, traditional image privacy protection methods cannot fully protect the privacy information in images. Therefore, many deep learning-based image privacy protection methods have emerged, such as k-same-net. This method clusters face features, then uses the cluster center as the average face feature vector, and then reconstructs the anonymized face image through a generation network, and uses this image to replace k images in the original cluster to achieve face image privacy protection.

[0005] The above existing technologies still have the following disadvantages: (1) From the perspective of privacy protection results, the prevention effect of traditional methods is poor and they cannot resist reliable image recognition software. And it may cause the phenomenon of information loss in the anonymized image. (2) Other methods such as k-same-Net have a too complex process, which is divided into multiple stages, and a suitable additional proxy dataset needs to be selected, and the whole process takes a relatively long time. (3) Some methods use methods such as GAN and k-means clustering to achieve image privacy protection, but the time complexity of these algorithms is too high, the overall number of model parameters is very large, and the whole anonymization process consumes too much computer resources. Summary of the Invention

[0006] The purpose of the present invention is to provide a lightweight image data privacy protection device and method to solve at least one of the technical problems in the above background technology.

[0007] To achieve the above purpose, the present invention adopts the following technical solutions:

[0008] On the one hand, the present invention provides a lightweight image data privacy protection device, including:

[0009] An image feature encoder module for efficiently capturing feature information from an image and converting it into a series of feature maps;

[0010] An image anonymization module for eliminating the privacy information contained in the feature map and generating an anonymized feature map;

[0011] An image generation module for reconstructing the anonymized feature map and finally generating an anonymized image.

[0012] Preferably, the image feature encoder module includes:

[0013] An inverse residual unit for capturing the spatial information in the feature map and reducing the number of model parameters;

[0014] An attention module for weighting the spatial regions and channels in the feature map to enhance the feature capture ability of the network;

[0015] The residual connection is used to construct feature maps of different levels and enhance the feature capture ability of the network.

[0016] Preferably, the image anonymization module includes:

[0017] The feature map grouping unit is used to divide the feature maps obtained by the image feature encoder module into several groups to meet the anonymization conditions;

[0018] The feature map anonymization unit is used to perform anonymization operations on the feature maps, average k feature maps, and eliminate the privacy information in the feature maps.

[0019] Preferably, the image generation module includes:

[0020] The decoding unit is used to synthesize the spatial information in the feature maps and adjust the number of channels of the feature maps;

[0021] The attention unit is used to weight the spatial regions and channels in the feature maps to enhance the feature capture ability of the network;

[0022] The upsampling unit is used to gradually restore the resolution of the average feature map and gradually reconstruct it into an anonymized image.

[0023] In a second aspect, the present invention provides an image data privacy protection method using the lightweight image data privacy protection device as described above, including:

[0024] Step 1: Select a suitable data set to pre-train the model and retain the model weights after pre-training;

[0025] Step 2: Preprocess the image data set that needs privacy protection;

[0026] Step 3: Output the preprocessed image data set into the encoder to generate a set of feature maps;

[0027] Step 4: Pass the feature maps generated in the previous step through the image anonymization module to generate anonymized feature maps;

[0028] Step 5: Input the anonymized feature maps into the image generation model to generate anonymized images;

[0029] Step 6: Select a suitable recognition model to verify the privacy protection effect and adjust the model according to the obtained results.

[0030] Advantages of the present invention: The number of algorithm parameters is greatly reduced, which is more conducive to deployment. At the same time, the computational complexity of the model, the training time, and the consumption of hardware are effectively reduced; through the designed image feature encoding module, the attention mechanism can be used to efficiently and accurately extract image features, improving the feature extraction ability of the model; using the greedy idea, the feature maps are grouped simply and efficiently to anonymize the image features, which takes less time compared to other methods such as clustering; the average image feature map can be efficiently reconstructed into an image without identity, and the reconstructed image not only realizes the privacy protection of the image but also retains some image information unrelated to identity.

[0031] The advantages of additional aspects of the present invention will be more clearly given in the following description part or understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0033] Figure 1 It is a schematic flowchart of the method for protecting the privacy of image data according to the embodiment of the present invention.

[0034] Figure 2 It is a functional principle block diagram of the model according to the embodiment of the present invention.

[0035] Figure 3 It is a schematic structural diagram of the Cblock according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The following details the embodiments of the present invention. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0037] Those skilled in the art of the present technology can understand that unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as the general understanding of those of ordinary skill in the art in the field to which the present invention belongs.

[0038] It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art and will not be interpreted with an idealized or overly formal meaning unless defined as here.

[0039] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or their groups.

[0040] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0041] To facilitate the understanding of the present invention, the present invention will be further explained below with specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0042] Those skilled in the art should understand that the drawings are only schematic diagrams of the embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0043] Embodiment 1

[0044] This Embodiment 1 provides a lightweight image data privacy protection device, including:

[0045] An efficient image feature encoder module, used to efficiently capture feature information from an image and convert it into a series of feature maps;

[0046] An image anonymization module, used to eliminate the privacy information contained in the image feature map and generate an anonymized feature map;

[0047] An image generation module, used to reconstruct the anonymized feature map and finally generate an anonymized image.

[0048] The efficient image feature encoder module specifically includes: an inverse residual module, used to efficiently capture the spatial information in the feature map and reduce the number of model parameters; an attention module, used to weight the spatial regions and channels in the feature map to enhance the feature capture ability of the network; a residual connection, used to construct feature maps of different levels and enhance the feature capture ability of the network.

[0049] The specific steps of the image anonymization module include: feature map grouping, which is used to divide the feature maps obtained from the efficient image feature encoder module into several groups to meet the conditions of k-anonymization; feature map anonymization, which is used to perform k-anonymization operations on the feature maps, average k feature maps, and eliminate the privacy information in the feature maps.

[0050] The specific steps of the image generation module include: a decoding module, which is used to synthesize the spatial information in the feature maps and adjust the number of channels of the feature maps; an attention module, which is used to weight the spatial regions and channels in the feature maps to enhance the feature capture ability of the network; an upsampling module, which is used to gradually restore the resolution of the average feature map and gradually reconstruct it into an anonymized image.

[0051] The working method of the lightweight image data privacy protection method and device includes the following steps:

[0052] Step 1: Select a suitable data set to pre-train the model and retain the model weights after pre-training;

[0053] Step 2: Preprocess the image data set that needs privacy protection;

[0054] Step 3: Output the preprocessed image data set into the encoder to generate a set of feature maps;

[0055] Step 4: Pass the feature maps generated in the previous step through the image anonymization module to generate anonymized feature maps;

[0056] Step 5: Input the anonymized feature maps into the image generation model to generate anonymized images;

[0057] Step 6: Select a suitable recognition model to verify the privacy protection effect and adjust the model according to the obtained results.

[0058] Embodiment 2

[0059] Such as Figures 1 to 3As shown in the figure, in the second embodiment, a lightweight image data privacy protection method and device are designed. The overall architecture of this method is based on the autoencoder and k-anonymization technology. First, a lightweight encoder module is designed using the inverted residual module and the attention mechanism to efficiently extract the feature map F from the input image dataset I. Secondly, a k-anonymization technology based on the greedy idea is used to divide the feature map F obtained in the first step into several groups, cluster them into feature map groups according to the shortest distance, with each group containing at least k feature maps, and calculate the average feature map in each group to eliminate the identity identification information in the feature map. Finally, a lightweight decoder is also constructed using the inverted residual module and the attention mechanism, and the anonymized images in each group are reconstructed through this decoder. This image can not only effectively eliminate the identity features of the original image and complete the task of privacy protection, but also retain other relevant attributes in the image, such as the expression and other information in the face image, so that the anonymized image can still be applied to related tasks.

[0060] S1: Model pre-training stage

[0061] Step 1-1: Use the existing benchmark similar image dataset to pre-train the entire encoder-decoder structure, retain the weights left by the pre-training, and do not consider the anonymization step in this stage. The purpose of the pre-training is to make the generated image as similar as possible to the original image.

[0062] S2: Input image preprocessing stage

[0063] Step 2-1: Obtain a set of images from the image dataset. The identifiers in this set of images must be different and there should be no duplicates.

[0064] Step 2-2: Perform a certain cropping on the images to make them of the same size, and appropriately use data augmentation operations such as random flipping and rotation.

[0065] S3: Image feature extraction stage

[0066] Step 3-1: Set various parameters of the image feature encoder, such as the number of training epochs, learning rate, network width, etc.

[0067] Step 3-2: Take the preprocessed image as the input, pass it through the pre-trained feature encoder, and obtain a set of feature maps F for representing the image through a series of convolution and other operations.

[0068] S4: Image feature anonymization stage

[0069] Step 4-1: According to the set k value and the image set I, take the obtained feature map set F and the parameter k as the input of this step, and this step returns an anonymized feature map set F based on I. a 。

[0070] Step 4-2: Assume that there are N feature maps in F in total. First, randomly select one feature map as a group. Using cosine similarity as the metric, gradually select the feature maps that are closest to this feature map and add them to this group until this group contains k feature maps.

[0071] Step 4-3: Repeat the above steps in a loop to divide F into N / k groups. If N is not divisible by k, add the remaining feature maps to the already divided groups. Therefore, each group contains at most 2k - 1 feature maps and at least k feature maps.

[0072] Step 4-4: In the first group, use k-anonymization to average the k image features.

[0073]

[0074] Keep repeating the above operations until a set of average face feature maps is finally obtained. where a = N / k.

[0075] S5: Image generation stage

[0076] Step 5-1: Use the set of average feature maps obtained in Step 3 as the input, and pass it through the pre-trained decoder to finally obtain a set of anonymized image set Y. i , j ∈ (1, a).

[0077] Step 5-2: Replace the k images in the original group j with the anonymized image y j Perform this operation for each group, and finally anonymize the entire image dataset to obtain the anonymized image set D.

[0078] S6: Algorithm performance analysis stage

[0079] Step 6-1: Select a suitable publicly available image dataset to be used as a benchmark for judging the algorithm performance.

[0080] Step 6-2: First, consider the attack strategies available to the attacker. The main purpose of the attacker is to re-identify the identity information of the images from the image data with the identity removed and verify the effectiveness of the final experiment.

[0081] Step 6-3: To verify whether the algorithm really plays a role in protecting privacy, after the k-anonymization process, use different attack strategies to verify the above two groups of images.

[0082] As shown in Table 1, the specific parameter details of the network structure in the model.

[0083] Table 1

[0084]

[0085] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0086] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0087] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0089] Although the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions disclosed in the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts should be covered within the protection scope of the present invention.

Claims

1. A lightweight image data privacy protection device, characterized in that, comprising: An image feature encoder module for efficiently capturing feature information from an image and converting it into a series of feature maps; An image anonymization module for eliminating the privacy information contained in the feature maps and generating anonymized feature maps; An image generation module for reconstructing the anonymized feature maps and finally generating anonymized images; The image feature encoder module includes: An inverse residual unit for capturing the spatial information in the feature maps and reducing the number of model parameters; An attention module for weighting the spatial regions and channels in the feature maps to enhance the network's feature capture ability; A residual connection for constructing feature maps of different levels and enhancing the network's feature capture ability; The image anonymization module includes: A feature map grouping unit for dividing the feature maps obtained by the image feature encoder module into several groups to meet the anonymization conditions; Feature map anonymization unit, which is used to perform anonymization operations on feature maps, average k feature maps, and eliminate privacy information in the feature maps; specifically, taking the obtained feature map set F and parameter k as inputs according to the set k value and image set I, an anonymized feature map set F based on I is obtained a ; Suppose there are N feature maps in F in total. First, randomly select one feature map as a group, use cosine similarity as the metric, and gradually select the feature maps closest to this feature map to join the group until the group contains k feature maps; repeat the above steps in a loop to divide F into N / k groups. If N cannot be divided evenly by k, then add the remaining feature maps to the already divided groups. Therefore, each group contains at most 2k - 1 feature maps and at least k feature maps; in the first group, use k-anonymization to average the k image features, and continuously repeat the above operations to finally obtain a group of average feature maps; The image generation module includes: A decoding unit for synthesizing the spatial information in the feature maps and adjusting the number of channels of the feature maps; An attention unit for weighting the spatial regions and channels in the feature maps to enhance the network's feature capture ability; An upsampling unit for gradually restoring the resolution of the average feature map and gradually reconstructing it into an anonymized image.

2. An image data privacy protection method using the lightweight image data privacy protection device as described in claim 1, characterized in that, comprising: Step 1: Select a suitable data set to pre-train the model and retain the model weights after pre-training; Step 2: Preprocess the image data set that needs privacy protection; Step 3: Output-encode the preprocessed image data set to generate a set of feature maps; Step 4: Pass the feature maps generated in the previous step through the image anonymization module to generate anonymized image feature maps; Step 5: Input the anonymized feature maps into the image generation model to generate anonymized images; Step 6: Select a suitable recognition model to verify the privacy protection effect and adjust the model according to the obtained results.

Images