Image data security processing method and device, computer equipment and storage medium
By performing pixel value conversion, random pixel flip and gradient iterative optimization on image data, veil data is generated, and the problem of taking into account both the privacy hiding and usability of image data in the prior art is solved, and the visual privacy information is effectively hidden and potential feature information is retained. Effect.
Patent Information
- Application Number
- CN202410887406.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-12-03
AI Technical Summary
When the prior art takes into account the availability of image data, it is difficult to effectively hide the visual privacy information of image data, resulting in the difference in availability between the encrypted image and the original image in actual scenes, and the person cannot be recognized normally.
Veil data is generated by performing pixel value conversion, random pixel flips and gradient iteration optimization operations on image data, which hides visual privacy information but retains potential feature information, and is suitable for deep learning model training and prediction.
It realizes that while taking into account the availability of image data, the visual privacy information of image data is effectively hidden. The generated veil data will not leak visual privacy information, and has potential feature information similar to the original image data, ensuring the availability of the deep learning model.
Smart Images

Figure CN119068069A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image data privacy security technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for secure processing of image data. Background Art
[0002] Deep learning models based on deep neural networks (DNNs) have achieved remarkable success in various disciplines. As we all know, deep learning models are data-driven, and the rapid development of DNNs has greatly benefited from the availability of open access datasets. High-quality datasets enable researchers to effectively verify the quality of their algorithms. However, most datasets contain private privacy information, such as facial data containing many biomedical feature attributes. Therefore, many privacy protection regulations, such as the EU General Data Protection Regulation (GDPR) and the US California Consumer Privacy Act (CCPA), require effective measures to protect the privacy security of data and ensure that deep learning service providers use user data legally and reasonably in a privacy-protecting manner. Among them, image data plays a key role in deep learning computer vision services, and most images contain personal sensitive privacy information, such as faces, license plates, etc. Specifically, facial data used for sharing to build face recognition models also leaks other biometric features such as gender and age.
[0003] There are currently methods for encrypting and protecting image data. The mainstream methods include the use of encrypted computing, fully homomorphic encryption and other technologies, such as RNS-KKS fully homomorphic encryption, pixel-based image encryption methods, Arnold scrambling encryption technology, etc. These methods encrypt the original image data, and then train and predict the model on the encrypted image data, but the model cannot normally recognize the original image data. For example, the RNS-KKS fully homomorphic encryption method can use specific technology to encrypt face images, and the encrypted images obtained will not leak any visual privacy information. However, the deep learning model trained using encrypted images can only recognize encrypted images and cannot predict the original image data. This results in completely different usability of encrypted images and original image data in real scenarios. One of the most direct real manifestations is that the face recognition system built using encrypted images cannot normally recognize the person in the actual scenario.
[0004] It can be seen that how to effectively hide the visual privacy information of image data while taking into account the availability of image data has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for secure processing of image data in response to the above technical problems.
[0006] According to a first aspect of an embodiment of the present application, a method for securely processing image data is provided, the method comprising: converting pixel values of first image data to be protected so that the pixel values of the first image data are converted from an original pixel value domain to a target pixel value domain suitable for pixel value processing of the first image data; performing a random pixel flipping operation on pixel values in the first image data that are in the target pixel value domain to obtain second image data after the random pixel flipping operation, wherein the second image data hides visual privacy information of the first image data, and the visual privacy information is privacy information perceptible at the human visual level in the first image data; based on the first image data, performing a gradient iterative optimization operation on the second image data to obtain third image data after gradient iterative optimization, wherein the third image data has second latent feature information similar to the first latent feature information of the first image data, and the second latent feature information is privacy information in the third image data used for deep learning model training and prediction; converting pixel values of the third image data so that the pixel values of the third image data are in the original pixel value domain, and determining that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0007] In one embodiment, the pixel values of the first image data to be protected are converted so that the pixel values of the first image data are converted from an original pixel value domain to a target pixel value domain suitable for pixel value processing of the first image data, including: based on a preset pixel value ratio, the pixel values of the first image data are reduced to obtain the reduced pixel values of the first image data; based on a preset mean and a preset variance, a normalization operation is performed on the reduced pixel values of the first image data to map the pixel values of the first image data from the original pixel value domain to the target pixel value domain.
[0008] In one embodiment, the random pixel flipping operation is performed on the pixel values in the first image data that are in the target pixel value range to obtain the second image data after the random pixel flipping operation, including: determining a random pixel flipping mask for the random pixel flipping operation based on a pre-created multidimensional matrix and a pre-set pixel flipping ratio, wherein the dimension of the multidimensional matrix is the same as the dimension of the first image data; based on the random pixel flipping mask, a random pixel flipping operation is performed on the pixel values in the first image data that are in the target pixel value range to obtain the second image data.
[0009] In one of the embodiments, determining a random pixel flip mask for the random pixel flip operation based on a pre-created multidimensional matrix and a pre-set pixel flip ratio includes: subtracting the pixel flip ratio from all values in the multidimensional matrix to obtain a subtracted multidimensional matrix; and normalizing positive values and negative values in the subtracted multidimensional matrix to obtain the random pixel flip mask.
[0010] In one embodiment, based on the random pixel flipping mask, a random pixel flipping operation is performed on the pixel values in the target pixel value domain in the first image data to obtain the second image data, including: performing a matrix dot multiplication operation on the random pixel flipping mask and the first image data to obtain a dot product result of the random pixel flipping mask and the first image data, and determining the dot product result as the second image data.
[0011] In one embodiment, the method of performing a gradient iterative optimization operation on the second image data based on the first image data to obtain third image data after gradient iterative optimization includes: a feature extraction step: determining that the second image data is the image data to be optimized, and performing potential feature extraction on the image data to be optimized through a potential feature extraction model to obtain a potential feature representation vector of the image data to be optimized; a determination step: determining that the first image data is the target image data, and determining the potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on the potential feature representation vector of the target image data, the pixel value of the target image data, the potential feature representation vector of the image data to be optimized, and the pixel value of the image data to be optimized; an optimization step: optimizing the image data to be optimized based on the potential feature similarity data and the pixel difference data through a stochastic gradient descent algorithm to obtain optimized image data; iteratively performing the feature extraction step, the determination step and the optimization step until the sum of the potential feature similarity data and the pixel difference data is less than a preset iteration termination threshold.
[0012] In one of the embodiments, the determining of potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on the potential feature characterization vector of the target image data, the pixel value of the target image data, the potential feature characterization vector of the image data to be optimized, and the pixel value of the image data to be optimized includes: determining the potential feature similarity data between the image data to be optimized and the target image data based on the potential feature characterization vector of the target image data, the potential feature characterization vector of the image data to be optimized, and the potential feature weight parameter; determining the pixel difference data between the image data to be optimized and the target image data based on the pixel value of the target image data, the pixel value of the image data to be optimized, and the pixel weight parameter.
[0013] In a second aspect, the present application also provides a device for securely processing image data. The device includes: a first conversion module, which is used to convert pixel values of the first image data to be protected, so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data; a first operation module, which is used to perform a random pixel flipping operation on the pixel values in the first image data that are in the target pixel value domain, so as to obtain second image data after the random pixel flipping operation, wherein the second image data hides the visual privacy information of the first image data, and the visual privacy information is the privacy information that can be perceived by human vision in the first image data; a second operation module, which is used to perform a gradient iterative optimization operation on the second image data based on the first image data, so as to obtain third image data after gradient iterative optimization, wherein the third image data has second potential feature information similar to the first potential feature information of the first image data, and the second potential feature information is the privacy information in the third image data used for deep learning model training and prediction; a second conversion module, which is used to convert the pixel values of the third image data, so that the pixel values of the third image data are in the original pixel value domain, and determine that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0014] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: converting the pixel values of the first image data to be protected so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data; performing a random pixel flipping operation on the pixel values in the first image data that are in the target pixel value domain to obtain the second image data after the random pixel flipping operation, wherein the second image data hides the visual privacy information of the first image data, and the visual privacy information is the privacy information perceptible at the human visual level in the first image data; based on the first image data, performing a gradient iterative optimization operation on the second image data to obtain the third image data after the gradient iterative optimization, wherein the third image data has second potential feature information similar to the first potential feature information of the first image data, and the second potential feature information is the privacy information in the third image data used for deep learning model training and prediction; converting the pixel values of the third image data so that the pixel values of the third image data are in the original pixel value domain, and determining that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0015] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented: converting the pixel values of the first image data to be protected so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data; performing a random pixel flipping operation on the pixel values in the first image data that are in the target pixel value domain to obtain the second image data after the random pixel flipping operation, wherein the second image data hides the visual privacy information of the first image data, and the visual privacy information is the privacy information perceptible at the human visual level in the first image data; based on the first image data, performing a gradient iterative optimization operation on the second image data to obtain the third image data after the gradient iterative optimization, wherein the third image data has second potential feature information similar to the first potential feature information of the first image data, and the second potential feature information is the privacy information in the third image data used for deep learning model training and prediction; converting the pixel values of the third image data so that the pixel values of the third image data are in the original pixel value domain, and determining that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0016] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, which implements the following steps when executed by a processor: converting the pixel values of the first image data to be protected so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data; performing a random pixel flipping operation on the pixel values in the first image data that are in the target pixel value domain to obtain the second image data after the random pixel flipping operation, wherein the second image data hides the visual privacy information of the first image data, and the visual privacy information is the privacy information perceptible at the human visual level in the first image data; based on the first image data, performing a gradient iterative optimization operation on the second image data to obtain the third image data after the gradient iterative optimization, wherein the third image data has second potential feature information similar to the first potential feature information of the first image data, and the second potential feature information is the privacy information in the third image data used for deep learning model training and prediction; converting the pixel values of the third image data so that the pixel values of the third image data are in the original pixel value domain, and determining that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0017] The above-mentioned image data security processing method, apparatus, computer device, storage medium and computer program product first convert the pixel values of the first image data to be protected, so that the pixel values of the first image data are converted from the original pixel value range to the target pixel value range suitable for pixel value processing of the first image data, which can improve the image data processing efficiency and reduce the calculation cost of the image data. Then, a random pixel flipping operation is performed on the pixel values in the first image data that are in the target pixel value range to obtain the second image data after the random pixel flipping operation, which can effectively hide the visual privacy information in the first image data. Then, based on the first image data, a gradient iterative optimization operation is performed on the second image data to obtain a gradient iterative optimization operation. The optimized third image data can constrain the second image data to have potential feature information similar to that of the first image data, and can effectively guarantee the availability of the second image data. Finally, the pixel values of the third image data are converted so that the pixel values of the third image data are within the original pixel value domain, and the third image data after the pixel value conversion is determined to be the veil data corresponding to the first image data. Compared with the first image data, the veil data will not leak the visual privacy information of the first image data, and has potential feature information similar to that of the first image data, so that the veil data has availability, and can effectively hide the visual privacy information of the image data while taking into account the availability of the image data. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0019] Figure 1 A flowchart of the steps of the method for securely processing image data provided by an embodiment of the present invention;
[0020] Figure 2 A schematic diagram of the definition and classification of privacy information of image data provided by an embodiment of the present invention;
[0021] Figure 3 A schematic diagram of visual privacy protection and its application provided by an embodiment of the present invention;
[0022] Figure 4 A schematic diagram of an experimental effect of visual privacy protection provided by an embodiment of the present invention;
[0023] Figure 5 A schematic diagram of the structure of a secure processing device for image data provided by an embodiment of the present invention;
[0024] Figure 6 An internal structure diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0026] In one embodiment, Figure 1 As shown, a method for securely processing image data is provided. In this embodiment, the method includes the following steps:
[0027] Step S101 , converting pixel values of first image data to be protected so that the pixel values of the first image data are converted from an original pixel value domain to a target pixel value domain suitable for pixel value processing of the first image data.
[0028] In some optional embodiments, the pixel values of the first image data to be protected are converted so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data, including: based on a preset pixel value ratio, the pixel values of the first image data are reduced to obtain the reduced pixel values of the first image data; based on a preset mean and a preset variance, a standardization operation is performed on the reduced pixel values of the first image data to map the pixel values of the first image data from the original pixel value domain to the target pixel value domain. The preset pixel value ratio, the preset mean and the preset variance can all be set by those skilled in the art according to actual needs, and this embodiment does not impose any limitation on this.
[0029] In a specific example, the original image data is converted to a specific domain to facilitate data processing. The specific domain conversion technology is mainly based on the existing idea of data standardization and normalization. Given any original image (H×W×C), C=1 for grayscale images and C=3 for RGB color images, and the range of all pixel values of the image is usually within [0, 255]. The mean and variance of the pixel values of the image are calculated respectively. The general standardization operation is to subtract the mean from all pixel values and then further divide by the variance to map the original larger pixel value to a new smaller pixel value space. However, due to the differences between different images, directly using this method will result in certain differences in the space to which different images are finally mapped. Therefore, all pixel values are first reduced by a ratio of 255, and in the subsequent standardization operation, the mean and variance are fixed to 0.5 at the same time, so that all image pixels can be mapped to the same pixel value domain to achieve data normalization. The final effect is that the image pixel value is converted from the [0, 255] value domain to the [-1, 1] value domain. Performing various data operations in the [-1, 1] value range can greatly improve processing efficiency and reduce computing overhead.
[0030] In this embodiment, the visual privacy protection technology framework mainly includes random pixel flipping technology and gradient iteration algorithm based on feature vectors. Before performing these two operations on the original image data, a specific conversion domain is designed to improve data processing efficiency and reduce computing costs. Based on the idea of image data normalization and standardization, the pixel value is converted from [0,255] to a specific domain of [-1,1], and then random pixel flipping and gradient iteration algorithm operations are performed in this domain.
[0031] Step S102: performing a random pixel flipping operation on pixel values in the first image data that are in the target pixel value range, so as to obtain second image data after the random pixel flipping operation.
[0032] In the prior art, there is no clear definition and specific guidance for privacy information in image data. Many data privacy protection works do not give a clear definition for privacy information in images, and do not give specific guidance on how to divide privacy. Some studies regard images as a privacy whole, that is, to fully protect the data; some studies regard sensitive visual information in images as the object of privacy protection; and some studies regard feature information in the data that is useful to the DNN model as the target of privacy protection. In the embodiment of the present application, the privacy information in the image data is determined, and a clear definition and classification guidance are given for the image privacy information. Specifically, based on the background of machine learning, the privacy information of image data is defined and classified into two categories: privacy information that is perceptible at the human visual level (human-perceptible info) and potential feature information that can be used for model training and prediction (DNN-perceptible info). Specifically as follows: 1) Privacy information that is perceptible at the human visual level. Privacy information that humans can directly obtain from images, such as visual privacy information such as faces, license plates, and phone numbers contained in images; 2) Potential feature information that can be used for model training and prediction. Compared with human recognition of object targets, the recognition information obtained by deep learning models from data is different. The private information in the data that can be obtained by DNN and used for model training and prediction is defined as potential features. Both types of private information can be used as objects for data privacy protection. The specific classification and definition are as follows: Figure 2 shown.
[0033] In this embodiment, the second image data hides visual privacy information of the first image data, where the visual privacy information is privacy information in the first image data that is perceptible at the human visual level.
[0034] In some optional embodiments, the random pixel flipping operation is performed on the pixel values in the first image data that are in the target pixel value range to obtain the second image data after the random pixel flipping operation, including: determining a random pixel flipping mask for the random pixel flipping operation based on a pre-created multidimensional matrix and a pre-set pixel flipping ratio, wherein the dimension of the multidimensional matrix is the same as the dimension of the first image data; based on the random pixel flipping mask, the random pixel flipping operation is performed on the pixel values in the first image data that are in the target pixel value range to obtain the second image data. Thus, through the random pixel flipping mask, the random pixel flipping operation can be effectively performed on the pixel values in the first image data that are in the target pixel value range to obtain the second image data.
[0035] In a specific example, when determining a random pixel flip mask for the random pixel flip operation based on a pre-created multidimensional matrix and a pre-set pixel flip ratio, the pixel flip ratio is subtracted from all values in the multidimensional matrix to obtain the subtracted multidimensional matrix; the positive values and negative values in the subtracted multidimensional matrix are normalized respectively to obtain the random pixel flip mask.
[0036] In a specific example, when a random pixel flipping operation is performed on pixel values in the target pixel value range in the first image data based on the random pixel flipping mask, a matrix dot multiplication operation is performed on the random pixel flipping mask and the first image data to obtain a dot product result of the random pixel flipping mask and the first image data, and the dot product result is determined to be the second image data.
[0037] In a specific example, in order to effectively hide the visual privacy information in the original image data, the random pixel flipping technology is designed and used on the original image data. First, a random flipping mask needs to be constructed. A multidimensional matrix is created and randomly initialized. The dimension of the matrix is the same as the dimension of the original image (H×W×C). A standard normal distribution U(0,1) is generated, and random sampling is performed from it. The values obtained by random sampling are used to reinitialize the multidimensional matrix. Then, a pixel flipping ratio p is defined, and its value range is between 0 and 1. The size of the flipping ratio determines the strength of the subsequent random pixel flipping. The maximum value 1 means that all pixel values are flipped, and the minimum value 0 means that all pixel values are not flipped. Then, in order to implement the pixel flipping operation, all values in the multidimensional matrix are subtracted from the flipping ratio value, and then the positive and negative values are normalized to 1 and -1 respectively, so that a random flipping mask is obtained. The random flipping mask is a multidimensional (H×W×C) matrix containing only 1 and -1 values. Finally, the original image pixel matrix and the random flip mask matrix are subjected to a matrix dot multiplication operation (i.e., the dot product result of two matrices of the same dimension is a new matrix of the same dimension, and the value of each position in this new matrix is equal to the product of the values of the corresponding positions in the two original matrices), thus completing the pixel random flipping operation. In the process of batch data processing, due to the strong randomness of sampling from the standard normal distribution U(0,1), the random flip mask for each original image is different and approximately unique. Through this random pixel flipping operation, an appropriate flip ratio p is selected (experimentally verified, generally p=0.5 can achieve a strong visual information hiding effect), which can effectively hide the visual privacy information of the original image data. At the same time, different visual hiding effects can be achieved by adjusting the size of the flip ratio.
[0038] Step S103: Based on the first image data, perform a gradient iterative optimization operation on the second image data to obtain third image data after gradient iterative optimization.
[0039] In this embodiment, the third image data has second potential feature information similar to the first potential feature information of the first image data, and the second potential feature information is privacy information in the third image data used for deep learning model training and prediction.
[0040] In some optional embodiments, when performing a gradient iterative optimization operation on the second image data based on the first image data, the feature extraction step: determining the second image data as the image data to be optimized, and extracting potential features from the image data to be optimized through a potential feature extraction model to obtain a potential feature representation vector of the image data to be optimized; determining step: determining the first image data as the target image data, and determining potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on the potential feature representation vector of the target image data, the pixel value of the target image data, the potential feature representation vector of the image data to be optimized, and the pixel value of the image data to be optimized; optimizing step: optimizing the image data to be optimized based on the potential feature similarity data and the pixel difference data through a stochastic gradient descent algorithm to obtain optimized image data; iteratively performing the feature extraction step, the determining step, and the optimizing step until the sum of the potential feature similarity data and the pixel difference data is less than a preset iteration termination threshold. In this way, the second image data can be constrained to have potential feature information similar to the potential feature information of the first image data, and the availability of the second image data can be effectively guaranteed.
[0041] In a specific example, the method of determining the potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on the potential feature characterization vector of the target image data, the pixel value of the target image data, the potential feature characterization vector of the image data to be optimized, and the pixel value of the image data to be optimized includes: determining the potential feature similarity data between the image data to be optimized and the target image data based on the potential feature characterization vector of the target image data, the potential feature characterization vector of the image data to be optimized, and the potential feature weight parameter; determining the pixel difference data between the image data to be optimized and the target image data based on the pixel value of the target image data, the pixel value of the image data to be optimized, and the pixel weight parameter. In this way, the potential feature similarity data and pixel difference data between the image data to be optimized and the target image data can be accurately determined.
[0042] In a specific example, a specially designed gradient iteration algorithm is applied to the image with random pixel flipping operation to constrain the goal of similarity of the potential features of the data used for DNN model training and prediction, that is, to ensure that the new and old data have the same availability for the model. Specifically, by constructing a specific optimization problem, that is, for a specific DNN model (usually a local proxy model F(·), which is trained on the original data), the image after random pixel flipping is used as the image to be optimized x ~ , take the original image as the target image x, and input the two images into the local proxy model to obtain the output feature vector of the last layer of the model. The optimization goal is to make the two vectors as similar as possible on the one hand, and to increase the difference in pixel values between the image to be optimized and the target image on the other hand. The similarity of feature vectors can ensure that the two images have similar potential feature information for the model, while the pixel difference can ensure that the optimized image is visually different from the target image. Specifically, the sum of the L1 norm between the two vectors and the negative L2 norm between the two image pixels is used as the loss function, and two weight parameters α and β are used to adjust the importance of these two parts, that is, Loss = α·||F(x ~ )-F(x)|| 1 +β·(-1)·||x ~ -x|| 2 . In this process, the output vector of the model for the data is regarded as the display representation of the data's available potential features for the model, while the pixel value represents its visual representation. The image to be optimized is continuously iteratively optimized through the stochastic gradient descent method SGD (the learning rate lr = 0.01 needs to be adjusted based on different experimental settings) (the maximum number of iterations T is set according to the image size, for example, CelebA sets T = 1500, and CIFAR10 sets T = 500). At the same time, if the Loss is less than a specific threshold (generally set to 0.1), it is used as the iteration termination condition, so that the potential feature gap between the image to be optimized and the target image is gradually narrowed, so that the potential features of the two images used for model training and reasoning are consistent. At the same time, for one input, the output vectors of different intermediate layers of the model can be selected as different optimization criteria to achieve different degrees of feature similarity.
[0043] In this embodiment, in order to achieve a balance between the availability and privacy of sensitive data in real-world scenarios, a visual privacy protection technology framework for image data is proposed. The specific process of the technical framework proposed in this embodiment is as follows: 1) First, it is clear that the goal of privacy protection is the visual privacy information in the image data. At the same time, in order to maintain the availability of the data, it is also necessary to ensure that the potential features of the data are consistent. 2) Secondly, a reasonable evaluation index is determined. The structural similarity index SSIM is used to measure the degree of visual privacy protection, and the model-specific output feature vector is used to measure the degree of similarity of the potential features of the data. 3) Then, a pixel random flipping technology is designed to hide the visual privacy information in the original image data. 4) Then, a gradient iterative algorithm based on feature vectors is proposed to constrain the generated privacy protection data to have similar potential feature information as the original image data. 5) Finally, the data finally generated is called veiled data. Compared with the original data, the veiled data does not leak visual privacy information and has similar potential feature information. This approach can alleviate the following problems: 1) Simple random pixel flipping or other technical means (such as homomorphic encryption) can achieve the purpose of protecting visual privacy information, but it also destroys the potential characteristics of the data to a certain extent, thereby destroying the availability of the data. 2) Using the gradient iteration algorithm, the veiled data is constrained to have potential feature information similar to the original image data, thereby solving the problem of data availability in existing work.
[0044] Step S104, converting the pixel values of the third image data so that the pixel values of the third image data are within the original pixel value range, and determining that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0045] The conversion operation of step S104 is the reverse operation of the conversion operation of step S101 and will not be described in detail here.
[0046] The following is a detailed introduction to the image visual privacy protection framework and application process. The core issue faced by this application is how to generate veiled data suitable for use and sharing without significantly reducing the performance of the DNN model. There are two tasks that need to be solved: protecting the visual privacy information of the original image data and maintaining the potential features used for DNN model training and prediction. The research goal is that the veiled data will not leak the visual privacy information of the original image data, and can be used to build and reason about the DNN model. More importantly, the trained model can efficiently recognize the original image data. Figure 3As shown, the image visual privacy protection framework and the application process are generally divided into two stages: (1) Veil data generation stage. In this stage, due to the need for visual privacy protection, the data owner can use the image visual privacy protection framework proposed in the present invention to generate new veil data from the original image data, and then share the veil data with a third party. (2) Veil data application stage. In this stage, the third party obtains the veil data from the data owner, and then uses the veil data to train the deep learning model to provide deep learning services. The trained deep learning service model can not only identify the veil data in the testing phase, but also effectively predict the original data.
[0047] Some specific implementation examples are provided for various data sets and deep learning models.
[0048] (1) Experimental setup
[0049] Seven image datasets that are widely used in the field of deep learning vision are set up, including four face datasets, BioID, ORL, LFW and CelebA, for face recognition tasks, and three image data, MNIST, CIFAR10 and CIFAR100, for image classification tasks. Among them, BioID includes 20 identities, a total of 1488 grayscale face images, and the image size is 384*286; ORL contains 11 identities, each identity includes 10 grayscale face images, the size is 92*112; LFW includes 5749 people, with a total of 13233 250*250 RGB face images; CelebA includes 10177 identities, including 202599 RGB images, the size is 178*218; MNIST contains 10 categories, a total of 60,000 grayscale images, the size is 28*28; CIFAR10 contains 10 categories, a total of 60,000 RGB images, the size is 32*32, and CIFAR100 contains 100 different categories.
[0050] At the same time, four classic deep learning models LeNet-5, AlexNet, VGG16 and VisionTransformer (VIT) are set. Among them, VGG16 is selected as the baseline model for BioID, LFW and CelebA, and LeNet-5 is selected as the baseline model for ORL; AlexNet is selected as the baseline model for MNIST and CIFAR10, and VIT is selected as the baseline model for CIFAR100.
[0051] For the training of the classification model, cross entropy can be selected as the loss function and stochastic gradient descent (SGD) can be used as the optimizer. For the generation of veil data, the flip ratio is generally set to 0.5, the hyperparameters α=0.999, β=0.001 and the maximum number of iterations in the gradient iteration optimization are 300, and the iteration termination threshold is set to 0.3. The details of different embodiments are as follows:
[0052]
[0053] Seven data sets and four types of classic deep learning models, each data set corresponds to dimension, category, quantity and model.
[0054] (2) Process
[0055] a. Build a baseline model. First, use the original data of different datasets to train the corresponding baseline model.
[0056] b. Veiled data generation. Afterwards, the visual privacy protection framework is used to generate veiled data from the original data.
[0057] c. Application. Use the veiled data instead of the original data to train the veiled model.
[0058] (3) Effect
[0059] Taking the CelebA and CIFAR10 datasets as examples, the patent of this invention was basically verified.
[0060] a. After the baseline model training is completed, the accuracy of CelebA is 87.96% and the accuracy of CIFAR10 is 85.99%.
[0061] b. Using the method proposed in the present invention, generate veil data corresponding to two data sets respectively, for example Figure 6 shown.
[0062] c. Use the VEIL data to build the VEIL model.
[0063] For CelebA, the veiled model achieves 87.09% accuracy on the veiled data and 83.87% accuracy on the original data.
[0064] For CIFAR10, the veiled model achieves 84.09% accuracy on the veiled data and 81.73% accuracy on the original data.
[0065] like Figure 4As shown, CelebA (left three columns) and CIFAR10 (right three columns). The first row is the original data, and the second row is the masked data.
[0066] In the above-mentioned method for securely processing image data, the pixel values of the first image data to be protected are first converted, so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data, which can improve the image data processing efficiency and reduce the calculation cost of the image data. Then, a random pixel flipping operation is performed on the pixel values in the first image data that are in the target pixel value domain to obtain the second image data after the random pixel flipping operation, which can effectively hide the visual privacy information in the first image data. Then, based on the first image data, a gradient iterative optimization operation is performed on the second image data to obtain the third image data after the gradient iterative optimization. It is capable of constraining the second image data to have potential feature information similar to that of the first image data, and can effectively guarantee the availability of the second image data. Finally, the pixel values of the third image data are converted so that the pixel values of the third image data are within the original pixel value domain, and the third image data after the pixel value conversion is determined to be the veil data corresponding to the first image data. Compared with the first image data, the veil data will not leak the visual privacy information of the first image data, and has potential feature information similar to that of the first image data, so that the veil data has availability. While taking into account the availability of the image data, the visual privacy information of the image data can be effectively hidden.
[0067] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0068] Based on the same inventive concept, the embodiment of the present application also provides a device for securely processing image data for implementing the method for securely processing image data involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the method above, so the specific limitations in the embodiments of one or more devices for securely processing image data provided below can refer to the limitations of the method for securely processing image data above, and will not be repeated here.
[0069] In one embodiment, Figure 5 As shown, a secure processing device for image data is provided, comprising: a first conversion module 501, a first operation module 502, a second operation module 503 and a second conversion module 504, wherein the first conversion module 501 is used to convert pixel values of first image data to be protected, so that the pixel values of the first image data are converted from the original pixel value domain to the target pixel value domain suitable for pixel value processing of the first image data; the first operation module 502 is used to perform a random pixel flipping operation on the pixel values in the first image data that are in the target pixel value domain, so as to obtain second image data after the random pixel flipping operation, wherein the second image data hides the visual privacy information of the first image data, and the visual privacy information is the The privacy information in the first image data that is perceptible at the human visual level; a second operation module 503, used to perform a gradient iterative optimization operation on the second image data based on the first image data to obtain third image data after gradient iterative optimization, wherein the third image data has second latent feature information similar to the first latent feature information of the first image data, and the second latent feature information is the privacy information in the third image data used for deep learning model training and prediction; a second conversion module 504, used to convert the pixel values of the third image data so that the pixel values of the third image data are in the original pixel value domain, and determine that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
[0070] In one embodiment, the first conversion module 501 is specifically used to: based on a preset pixel value ratio, reduce the pixel values of the first image data to obtain the reduced pixel values of the first image data; based on a preset mean and a preset variance, perform a standardization operation on the reduced pixel values of the first image data to map the pixel values of the first image data from the original pixel value domain to the target pixel value domain.
[0071] In one embodiment, the first operation module 502 is specifically used to: determine a random pixel flipping mask used for the random pixel flipping operation based on a pre-created multidimensional matrix and a pre-set pixel flipping ratio, wherein the dimension of the multidimensional matrix is the same as the dimension of the first image data; based on the random pixel flipping mask, perform a random pixel flipping operation on pixel values in the first image data that are in the target pixel value range to obtain the second image data.
[0072] In one embodiment, the first operation module 502 is specifically used to: subtract the pixel flip ratio from all values in the multidimensional matrix to obtain the subtracted multidimensional matrix; and normalize the positive values and negative values in the subtracted multidimensional matrix to obtain the random pixel flip mask.
[0073] In one embodiment, the first operation module 502 is specifically used to: perform a matrix dot product operation on the random pixel flip mask and the first image data to obtain a dot product result of the random pixel flip mask and the first image data, and determine the dot product result as the second image data.
[0074] In one embodiment, the second operation module 503 is specifically used for: a feature extraction step: determining that the second image data is the image data to be optimized, and performing potential feature extraction on the image data to be optimized through a potential feature extraction model to obtain a potential feature representation vector of the image data to be optimized; a determination step: determining that the first image data is the target image data, and determining the potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on the potential feature representation vector of the target image data, the pixel value of the target image data, the potential feature representation vector of the image data to be optimized, and the pixel value of the image data to be optimized; an optimization step: optimizing the image data to be optimized based on the potential feature similarity data and the pixel difference data through a stochastic gradient descent algorithm to obtain optimized image data; iteratively executing the feature extraction step, the determination step and the optimization step until the sum of the potential feature similarity data and the pixel difference data is less than a preset iteration termination threshold.
[0075] In one embodiment, the second operation module 503 is specifically used to: determine the potential feature similarity data between the image data to be optimized and the target image data based on the potential feature characterization vector of the target image data, the potential feature characterization vector of the image data to be optimized, and the potential feature weight parameter; determine the pixel difference data between the image data to be optimized and the target image data based on the pixel value of the target image data, the pixel value of the image data to be optimized, and the pixel weight parameter.
[0076] Each module in the above-mentioned image data security processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.
[0077] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for securely processing image data is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0078] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0079] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0080] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0081] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0084] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0085] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for securely processing image data, characterized in that: The method comprises: Converting pixel values of the first image data to be protected so that the pixel values of the first image data are converted from an original pixel value domain to a target pixel value domain suitable for pixel value processing of the first image data; Performing a random pixel flipping operation on pixel values in the first image data that are in the target pixel value range to obtain second image data after the random pixel flipping operation, wherein the second image data hides visual privacy information of the first image data, and the visual privacy information is privacy information in the first image data that is perceptible to human vision; Based on the first image data, performing a gradient iterative optimization operation on the second image data to obtain third image data after gradient iterative optimization, wherein the third image data has second latent feature information similar to the first latent feature information of the first image data, and the second latent feature information is private information in the third image data used for deep learning model training and prediction; The pixel values of the third image data are converted so that the pixel values of the third image data are within the original pixel value domain, and the third image data after the pixel value conversion is determined to be the veil data corresponding to the first image data.
2. The method for securely processing image data according to claim 1, characterized in that: The step of converting the pixel values of the first image data to be protected so that the pixel values of the first image data are converted from an original pixel value range to a target pixel value range suitable for pixel value processing of the first image data includes: Based on a preset pixel value ratio, reducing the pixel values of the first image data to obtain reduced pixel values of the first image data; Based on a preset mean and a preset variance, a normalization operation is performed on the reduced pixel values of the first image data to map the pixel values of the first image data from the original pixel value domain to the target pixel value domain.
3. The method for securely processing image data according to claim 1, characterized in that: The performing a random pixel flipping operation on the pixel values in the first image data that are in the target pixel value range to obtain second image data after the random pixel flipping operation comprises: Determining a random pixel flipping mask for the random pixel flipping operation based on a pre-created multi-dimensional matrix and a pre-set pixel flipping ratio, wherein the dimension of the multi-dimensional matrix is the same as the dimension of the first image data; Based on the random pixel flipping mask, a random pixel flipping operation is performed on pixel values in the first image data that are in the target pixel value domain to obtain the second image data.
4. The method for securely processing image data according to claim 3, characterized in that: The step of determining a random pixel flipping mask for the random pixel flipping operation based on a pre-created multi-dimensional matrix and a pre-set pixel flipping ratio includes: Subtract the pixel flip ratio from all values in the multidimensional matrix to obtain a subtracted multidimensional matrix; The positive values and negative values in the subtracted multidimensional matrix are respectively normalized to obtain the random pixel flip mask.
5. The method for securely processing image data according to claim 3, characterized in that: The step of performing a random pixel flipping operation on pixel values in the first image data that are in the target pixel value range based on the random pixel flipping mask to obtain the second image data comprises: A matrix dot multiplication operation is performed on the random pixel flipping mask and the first image data to obtain a dot product result of the random pixel flipping mask and the first image data, and the dot product result is determined to be the second image data.
6. The method for securely processing image data according to claim 1, characterized in that: The step of performing a gradient iterative optimization operation on the second image data based on the first image data to obtain third image data after gradient iterative optimization includes: Feature extraction step: determining that the second image data is the image data to be optimized, and performing potential feature extraction on the image data to be optimized through a potential feature extraction model to obtain a potential feature representation vector of the image data to be optimized; Determining step: determining that the first image data is target image data, and determining potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on a potential feature characterization vector of the target image data, a pixel value of the target image data, a potential feature characterization vector of the image data to be optimized, and a pixel value of the image data to be optimized; Optimization step: optimizing the image data to be optimized based on the potential feature similarity data and the pixel difference data by a stochastic gradient descent algorithm to obtain optimized image data; The feature extraction step, the determination step and the optimization step are iteratively performed until the sum of the potential feature similarity data and the pixel difference data is less than a preset iteration termination threshold.
7. The method for securely processing image data according to claim 6, characterized in that: The determining of potential feature similarity data and pixel difference data between the image data to be optimized and the target image data based on the potential feature representation vector of the target image data, the pixel value of the target image data, the potential feature representation vector of the image data to be optimized, and the pixel value of the image data to be optimized includes: Determining potential feature similarity data between the image data to be optimized and the target image data based on the potential feature representation vector of the target image data, the potential feature representation vector of the image data to be optimized, and a potential feature weight parameter; Based on the pixel values of the target image data, the pixel values of the image data to be optimized, and the pixel weight parameters, pixel difference data between the image data to be optimized and the target image data is determined.
8. A secure processing device for image data, characterized in that: The device comprises: A first conversion module, used for converting pixel values of the first image data to be protected, so that the pixel values of the first image data are converted from an original pixel value domain to a target pixel value domain suitable for pixel value processing of the first image data; A first operation module is configured to perform a random pixel flipping operation on pixel values in the first image data that are in the target pixel value range, so as to obtain second image data after the random pixel flipping operation, wherein the second image data hides visual privacy information of the first image data, and the visual privacy information is privacy information in the first image data that is perceptible to human vision; a second operation module, configured to perform a gradient iterative optimization operation on the second image data based on the first image data to obtain third image data after gradient iterative optimization, wherein the third image data has second potential feature information similar to the first potential feature information of the first image data, and the second potential feature information is privacy information in the third image data used for deep learning model training and prediction; The second conversion module is used to convert the pixel values of the third image data so that the pixel values of the third image data are within the original pixel value range, and determine that the third image data after the pixel value conversion is the veil data corresponding to the first image data.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
