Self-sovereign approach method and apparatus for privacy preserving image generation and robust classification

Abstract

A self-sovereign approach method and apparatus for privacy preserving image generation and robust classification are disclosed. The self-sovereign approach method for privacy preserving image generation and robust classification, according to one embodiment of the present invention, may comprise the steps of: denoising a first differential privacy (DP) noise from a DP image so as to restore same to the original image; recognizing a sensitive object in the image of the restored original according to privacy preferences of a data owner; and adding a second DP noise only to the recognized sensitive object so as to generate a privacy-preserving (PP) image.

Description

A self-sovereign approach and device for privacy-preserving image generation and robust classification.

[0001] The present invention relates to a self-sovereign approach and device for generating privacy-preserving images and for robust classification, which generates privacy-preserving images based on the privacy preferences of data owners and effectively classifies images by utilizing the generated privacy-preserving images.

[0002] Publication No. 10-2023-0070879 (May 23, 2023) "Privacy Area De-identification and Restoration Device, Privacy Area De-identification and Restoration Method"

[0003] Publication number 10-2023-0070879 relates to a privacy area de-identification and restoration device and a privacy area de-identification and restoration method, which can significantly reduce the amount of additional data required for restoration after de-identifying a privacy area.

[0004] Registration No. 10-2486493 (January 4, 2023) "DCC conversion method and device for preserving privacy of learning image data"

[0005] Registration No. 10-2486493 relates to a DCC conversion method and device for preserving the privacy of learning image data, in order to reduce the degree of accuracy loss, which was a problem of conventional image conversion techniques, and to preserve the privacy of learning data.

[0006] Registration No. 10-2123248 (June 10, 2020) "Facial Recognition-Based Real-Time Image Processing System for Privacy Protection"

[0007] Registration number 10-2123248 discloses a real-time image processing system based on facial recognition that de-identifies the faces of people who have not consented to being photographed in order to prevent privacy violations by protecting their portrait rights.

[0008] Data is essential in modern society, and because data is essential, machine learning and deep learning algorithms are continuously developing.

[0009] However, advancements in machine learning and deep learning algorithms are simultaneously raising concerns about data privacy.

[0010] Differential Privacy (DP) was proposed as a mathematical approach to privacy to address concerns about data privacy.

[0011] Differential privacy limits information leakage based on the privacy variable ε and the relaxation variable δ.

[0012] The basic principle of differential privacy is to add DP noise as long as it does not distort the overall distribution of the data.

[0013] In particular, for image data, differential privacy can be implemented mainly through two strategies.

[0014] The first strategy for implementing differential privacy is to generate new images using generative models (GAN, VAE, diffusion models, etc.) that incorporate differential privacy features.

[0015] A second strategy for implementing differential privacy is to utilize image perturbation techniques that add noise to image pixels.

[0016] Nonetheless, existing strategies for implementing differential privacy have several major limitations.

[0017] First, existing strategies may degrade the performance of computer vision tasks based on perturbed or generated images.

[0018] Second, existing strategies fail to reflect the privacy preferences of data owners. Consequently, if the entire image is deemed sensitive, all pixels in the image are perturbed.

[0019] Therefore, there is an urgent need to implement improved differential privacy that reflects the privacy preferences of data owners and adds noise only to specific areas that data owners are sensitive to.

[0020] An embodiment of the present invention aims to provide a self-sovereign approach and device for privacy-preserving image generation and robust classification, which efficiently generates privacy-preserving images that precisely hide only specific sensitive objects without distorting the entire image.

[0021] Additionally, an embodiment of the present invention aims to enable data owners to designate specific and important sensitive information they wish to protect.

[0022] In addition, an embodiment of the present invention aims to accurately classify an image even if DP noise is included in a wide area.

[0023] Additionally, embodiments of the present invention aim to enhance the resistance of a model to model inversion attacks aimed at reconstructing sensitive objects of an image, since sensitive objects are excluded from the learning and prediction processes.

[0024] According to one embodiment of the present invention, a self-sovereign approach for privacy-preserving image generation and robust classification may include the steps of: denoising a first DP noise in a DP (Differential Privacy) image to restore the original image; recognizing a sensitive object in the restored original image according to a privacy preference of a data owner; and adding a second DP noise only to the recognized sensitive object to generate a privacy-preserving (PP) image.

[0025] In addition, a self-sovereign access device for privacy-preserving image generation and robust classification according to an embodiment of the present invention may be configured to include a privacy-preserving image generation module that denoises first DP noise in a DP (Differential Privacy) image to restore the original image, recognizes a sensitive object in the restored original image according to a privacy preference of a data owner, and adds second DP noise only to the recognized sensitive object to create a privacy-preserving (PP) image.

[0026] According to one embodiment of the present invention, a self-sovereign approach and device for privacy-preserving image generation and robust classification can be provided, which efficiently generates privacy-preserving images that precisely hide only specific sensitive objects without distorting the entire image.

[0027] Additionally, according to one embodiment of the present invention, it is possible to enable a data owner to designate specific and important sensitive information that he or she wishes to protect.

[0028] In addition, according to one embodiment of the present invention, even if DP noise is included in a wide area, an image can be accurately classified.

[0029] In addition, according to one embodiment of the present invention, since sensitive objects are excluded from the learning and prediction process, the resistance of a model to model inversion attacks aimed at reconstructing sensitive objects of an image can be strengthened.

[0030] FIG. 1 is a block diagram illustrating a configuration of a self-sovereign access device for privacy-preserving image generation and robust classification according to one embodiment of the present invention.

[0031] Figure 2 is a diagram visually showing the overall structure and operation process of the present invention.

[0032] Figure 3 is a diagram illustrating the process of generating a PP image.

[0033] Figure 4 is a diagram illustrating the process of generating a binary mask.

[0034] Figure 5 is a diagram explaining the process of extracting features from an image.

[0035] Figure 6 is a diagram for explaining the process of image classification.

[0036] Figure 7 is a diagram illustrating the overall operation of the self-sovereign access device.

[0037] FIG. 8 is a flowchart illustrating a self-sovereign approach for privacy-preserving image generation and robust classification according to one embodiment of the present invention.

[0038] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0039] The terms used in the examples are for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0040] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0041] In addition, when describing with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing embodiments, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

[0042] FIG. 1 is a block diagram illustrating a configuration of a self-sovereign access device for privacy-preserving image generation and robust classification according to one embodiment of the present invention.

[0043] Referring to FIG. 1, a self-sovereign access device (hereinafter, abbreviated as 'self-sovereign access device', 100) for privacy-preserving image generation and strong classification according to one embodiment of the present invention may be configured to include a preprocessing module (110), a privacy-preserving image generation module (120), and a classification module (130).

[0044] First, the privacy-preserving image generation module (120) denoises the first DP noise in the DP (Differential Privacy) image and restores it to the original image. In other words, the privacy-preserving image generation module (120) can remove the first DP noise from a DP image that is perturbed by noise and thus cannot recognize the original, thereby restoring the DP image to the original image.

[0045] A DP image can refer to an image that protects the privacy of data owners by quantitatively modeling privacy.

[0046] The DP image can be produced by the preprocessing module (110).

[0047] The preprocessing module (110) can prepare the DP image by perturbing the entire image with the first DP noise of the differential distribution, satisfying Equation 1.

[0048] [Formula 1]

[0049]

[0050] Here, the above △f is a sensitivity of up to 255 of the pixel value defined as the maximum value for a color image, and the above ε and δ may be privacy parameters set by the data owner.

[0051] That is, the preprocessing module (110) can perturb an image owned by a data owner, including a sensitive object, with DP noise having a differential distribution to prevent the sensitive object from being exposed to the outside.

[0052] The DP image produced by the preprocessing module (110) can mask the entire image, including the sensitive object, with the first DP noise so that the sensitive object is not revealed in the image itself.

[0053] Additionally, the privacy-preserving image generation module (120) recognizes sensitive objects within the restored original image, based on the data owner's privacy preferences. That is, the privacy-preserving image generation module (120) can identify specific objects designated by the data owner as objects to be preferentially hidden in the restored image.

[0054] Here, a sensitive object may be an object that the data owner designates based on his or her subjective judgment, as it is expected that privacy will be seriously violated if exposed externally.

[0055] In the present invention, a sensitive object can be designated by a preprocessing module (110) based on a Self-Sovereign Identity (SSI) profile.

[0056] The preprocessing module (110) may define an object designated by the data owner among multiple objects constituting the SSI (Self-Sovereign Identity) profile as the sensitive object. That is, the preprocessing module (110) may determine, in the SSI profile, a specific object designated by the data owner as an object to be hidden according to his / her own priority as a sensitive object.

[0057] Here, an SSI profile can be a data owner profile created in a self-sovereign identity environment. SSI can refer to an identity management model and implementation technology that empowers individuals to control the information they use to prove their identity on websites, services, and applications.

[0058] Additionally, the preprocessing module (110) can set the privacy preference, which is differentiated according to the order in which the data is defined, to each sensitive object. That is, the preprocessing module (110) can define an object as a sensitive object by granting it a relatively higher privacy preference the earlier the data owner designates the object.

[0059] Accordingly, the privacy-preserving image generation module (120) can sequentially recognize the sensitive object with the highest privacy preference. That is, the privacy-preserving image generation module (120) can sequentially recognize the sensitive object in order of highest privacy preference among multiple objects or sensitive objects in the restored original image.

[0060] Thereafter, the privacy-preserving image generation module (120) generates a PP (privacy-preserving) image by adding a second DP noise only to the recognized sensitive object. That is, the privacy-preserving image generation module (120) can play a role in generating a PP image in which only the sensitive object is obscured by adding a second DP noise to an area where a specific sensitive object recognized according to privacy preference is located within the reduced image.

[0061] In generating a PP image, the privacy-preserving image generation module (120) can generate a PP image by adding a second DP noise of a Laplace distribution to a recognized sensitive object.

[0062] The privacy-preserving image generation module (120) can generate the PP image by masking the sensitive object with the second DP noise of the Laplace distribution, satisfying Equation 2.

[0063] [Formula 2]

[0064]

[0065] The second DP noise of the Laplace distribution has a lower ε value than the first DP noise of the differential distribution, which makes the noise estimation and denoising process for the related generated PP image more difficult and allows for more robust protection of sensitive objects from external exposure.

[0066] According to an embodiment, the privacy-preserving image generation module (120) may generate a binary mask for distinguishing an area in the PP image where the second DP noise is masked, along with the generation of the PP image.

[0067] That is, the privacy-preserving image generation module (120) can generate a binary mask that distinguishes the second DP noise by making the pixels in the area where the second DP noise is added in the PP image have a value of 1 (white) and making the pixels in other areas have a value of 0 (black).

[0068] The above binary mask is used in subsequent image classification to identify pixels in the area to which the second DP noise has been added.

[0069] After the PP image is generated, the self-sovereign access device (100) can fuse the features of the DP image and the features of the PP image, and accurately classify the PP image including the second DP image through the fused features.

[0070] To this end, the self-sovereign access device (100) may be configured to further include a classification module (130).

[0071] That is, the classification module (130) can input the DP image and the PP image into a custom image classification model and output a DP feature map and a PP feature map from the custom image classification model.

[0072] The classification module (130) can create a custom image classification model that can extract key features of an image by utilizing various known or existing images.

[0073] Additionally, the classification module (130) can provide an enhanced image classification function through feature fusion and exclusion of areas to which second DP noise has been added.

[0074] The custom image classification model can extract features from each of the DP image and the PP image input by the classification module (130).

[0075] That is, a custom image classification model can learn about DP images to extract DP features, and learn about PP images to extract PP features.

[0076] In addition, the custom image classification model can produce the DP feature map and the PP feature map with the attention readjusted for each layer by adjusting the attention according to the importance of the feature by utilizing the attention mechanism.

[0077] That is, the custom image classification model can adjust and assign attention to each feature by considering the importance of each feature extracted through the squeeze and excitation (SE) block according to the above-mentioned attention mechanism, and can create a DP feature map and a PP feature map by matrixizing multiple features to which each adjusted attention is assigned.

[0078] Here, attention mechanisms can be used to improve the performance of neural networks by allowing the model to focus on the most important input data while generating predictions. This can be accomplished by weighting input data so that the model prioritizes some input attributes over others.

[0079] Thereafter, the classification module (130) can apply the binary mask to remove the second DP noise from the DP feature map and the PP feature map, and then fuse the features by weighted combining the DP feature map and the PP feature map. That is, the classification module (130) can apply the binary mask to each of the DP feature map and the PP feature map to distinguish the second DP noise from these feature maps, and combine only the remaining portions of the feature maps in which the distinguished second DP noise is not involved, thereby fusion of the features.

[0080] Weighted combining can mean combining feature maps in a state where a relatively large weight is given to a feature map that receives more attention depending on the implementation environment, through the preceding attention mechanism.

[0081] In addition, the classification module (130) can classify and identify the PP image using the fused features. That is, the classification module (130) can classify the image by layer classifying the fused features, thereby identifying what the generated PP image is, and output the identified result to the outside.

[0082] According to one embodiment of the present invention, a self-sovereign approach and device for privacy-preserving image generation and robust classification can be provided, which efficiently generates privacy-preserving images that precisely hide only specific sensitive objects without distorting the entire image.

[0083] Additionally, according to one embodiment of the present invention, it is possible to enable a data owner to designate specific and important sensitive information that he or she wishes to protect.

[0084] In addition, according to one embodiment of the present invention, even if DP noise is included in a wide area, an image can be accurately classified.

[0085] In addition, according to one embodiment of the present invention, since sensitive objects are excluded from the learning and prediction process, the resistance of a model to model inversion attacks aimed at reconstructing sensitive objects of an image can be strengthened.

[0086] Existing implementations of differential privacy have negatively impacted image processing performance while ignoring the privacy preferences of data owners.

[0087] To solve this problem, the self-sovereign access device (100) of the present invention can generate a privacy-preserving image centered on the privacy preference of the data owner, and can effectively classify the image by utilizing the generated privacy-preserving image.

[0088] The main technical principles of the self-sovereign access device (100) are as follows.

[0089] A self-sovereign access device (100) can be designed to enable data owners to strengthen their control over their data.

[0090] In particular, the self-sovereign access device (100) can generate a privacy-preserving image that protects only specific objects from the image according to the privacy preferences of the data owner.

[0091] To this end, the self-sovereign access device (100) can utilize the concept of 'self-sovereign identity (SSI)' to enable the data owner to designate an object subject to privacy protection (sensitive object) through his / her SSI profile.

[0092] The self-sovereign access device (100) generates a privacy-preserving image by adding noise only to a designated privacy-protected object within an image, thereby protecting only sensitive objects without distorting the entire image through the generated privacy-preserving image.

[0093] Privacy-preserving images are difficult to guarantee the same performance as the original images because of information loss during the denoising process and DP noise that causes confusion in deep learning models.

[0094] To solve these problems, the self-sovereign access device (100) can create a custom image classification model that utilizes various images (privacy-preserving images, differential privacy images, binary masks, etc.).

[0095] The self-sovereign access device (100) can implement a preprocessing step, a privacy-preserving image generation step, and an image classification step using a custom image classification model.

[0096] In the preprocessing step, the self-sovereign access device (100) can set the privacy preference of the data owner and set the DP noise to be added to the image.

[0097] In the privacy-preserving image generation step, the self-sovereign access device (100) can denoise an input PD image, recognize a sensitive object in the denoised image according to a privacy preference, and add DP noise to the recognized sensitive object to generate a privacy-preserving image (PP image).

[0098] In the image classification step using a custom image classification model, the self-sovereign access device (100) can provide an enhanced image classification function through extracting key features of an image, excluding an area to which DP noise has been added from the extracted features, and feature fusion by utilizing various images in the custom image classification model.

[0099] Figure 2 is a diagram visually showing the overall structure and operation process of the present invention.

[0100] The self-sovereign access device (100) can be performed by including a preprocessing step (210, 220), a privacy-preserving image generation step (230, 240), and an image classification step (250, 260, 270) using a custom image classification model.

[0101] In step 210 of the preprocessing step, the self-sovereign access device (100) can set the privacy preference of the data owner by defining sensitive objects within the SSI Profile (SSI Profile configuration).

[0102] In step 220 of the preprocessing step, the self-sovereign access device (100) can set the first DP noise of the differential distribution and generate a DP (Differential Privacy) image with the DP noise added to the entire image (Generate DP Image).

[0103] In step 230 of the privacy-preserving image generation step, the self-sovereign access device (100) denoises the DP image to restore it to the original image, and can recognize the location of a sensitive object within the restored original image according to the privacy preference set in the SSI profile (Denoise & Locate Sensitive Object).

[0104] In step 240 of the privacy-preserving image generation step, the self-sovereign access device (100) can hide sensitive objects by generating a PP (privacy-preserving) image by adding a second DP noise of a Laplace distribution only to the recognized sensitive objects (Conceal Sensitive Object by Adding DP Noise).

[0105] In step 250 of the image classification step, the self-sovereign access device (100) can extract features from each of the DP image and the PP image using a custom image classification model, and matrix-ize the extracted features to create a DP feature map and a PP feature map (Feature Extraction from DP & PP image).

[0106] In step 260 of the image classification step, the self-sovereign access device (100) can utilize a binary mask that distinguishes an area in the PP image to which DP noise has been added to exclude a noisy path (Exclude Noisy Path).

[0107] In step 270 of the image classification step, the self-sovereign access device (100) applies a binary mask to the DP feature map and the PP feature map, distinguishes DP noise, and then weights and combines the features to fuse the features (Feature fusion) and provide an image classification function (Weighted Combination).

[0108] As shown in FIG. 2, the self-sovereign access device (100) can perform a preprocessing step by a preprocessing module, a privacy-preserving image generation step by a preservation image generation module, and an image classification step using a custom image classification model by a classification module.

[0109] The preprocessing module can set privacy preferences by defining sensitive objects within the data owner's SSI profile.

[0110] The preprocessing module can set privacy preferences based on a list of sensitive objects specified by the data owner in the SSI profile.

[0111] The preprocessing module can extend the list of sensitive objects specified by the data owner to include objects that are inherently sensitive, such as personal identifiable information (PII) and personal health information (PHI).

[0112] The preprocessing module can generate a DP image by adding the first DP noise of differential distribution to the entire image of the data owner according to the Gaussian differential privacy mechanism expressed in Equation 1.

[0113] Equation 1 can express the Gaussian differential privacy mechanism.

[0114] [Formula 1]

[0115]

[0116] Here, △f represents the sensitivity of up to 255 of the pixel value defined as the maximum value for a color image, and ε and δ can represent privacy parameters set by the data owner.

[0117] The privacy-preserving image generation module can receive an SSI profile with a sensitive object specified and a privacy preference set, and a DP image as input.

[0118] The privacy-preserving image generation module can denoise the DP image to restore the original image.

[0119] The privacy-preserving image generation module can estimate the distribution of the first DP noise and perform denoising based on the premise that the first DP noise in the DP image follows a differential distribution.

[0120] The privacy-preserving image generation module can perform denoising, for example, using DnCNN. DnCNN is a denoising model based on a residual learning strategy that can effectively learn from Gaussian noise with various σ values.

[0121] The privacy-preserving image generation module can recognize sensitive objects in the restored original image according to the privacy preference set in the SSI profile, and generate a PP image by adding second DP noise of Laplace distribution only to the recognized sensitive objects.

[0122] The privacy-preserving image generation module can detect sensitive objects by utilizing the pre-trained Faster R-CNN feature detection module.

[0123] The privacy-preserving image generation module can be fine-tuned for the Faster R-CNN feature detection module to accurately detect sensitive objects determined by the SSI profile.

[0124] The privacy-preserving image generation module can accurately detect sensitive objects within an image by inputting the original image restored through denoising into the fine-tuned Faster R-CNN feature detection module.

[0125] The privacy-preserving image generation module can hide the sensitive object by perturbing the image within the bounding box surrounding the detected sensitive object with a second DP noise of a Laplace distribution with a low ε value as the sensitive object is detected.

[0126] The privacy-preserving image generation module can generate a PP image having heavy tails characteristics by adding second DP noise of a Laplace distribution, compared to adding first DP noise of a differential distribution, and can make the noise estimation and denoising process for the PP image more difficult.

[0127] Equation 2 can express the Laplace difference privacy mechanism.

[0128] [Formula 2]

[0129]

[0130] Figure 3 is a diagram illustrating the process of generating a PP image.

[0131] In step 310, the self-sovereign access device (100) receives the original image after denoising the DP image.

[0132] In step 320, the self-sovereign access device (100) can select the i-th object from the object list of the SSI profile of the data owner.

[0133] In step 330, the self-sovereign access device (100) can determine whether i of the selected ith object is less than the total number n of the object list.

[0134] If i is not less than n (No direction in step 330), the self-sovereign access device (100) can terminate the operation.

[0135] If i is less than n (Yes in step 330), then in step 340, the self-sovereign access device (100) can determine whether the i-th object is a sensitive object according to the privacy preference set in the SSI profile.

[0136] If it is not a sensitive object (No direction in step 340), the self-sovereign access device (100) can move to step 360.

[0137] If it is a sensitive object (Yes direction in step 340), in step 350, the self-sovereign access device (100) can generate a PP image by adding a second DP noise of Laplace distribution to the corresponding area (Bbox(Di)) of the image corresponding to the i-th object.

[0138] At step 360, the self-sovereign access device (100) can select the (i+1)th object from the object list by adding 1 to i and then returning to step 330.

[0139] In Fig. 3, the process of detecting a sensitive object in an image and hiding the detected sensitive object is described in detail.

[0140] The self-sovereign access device (100) can detect a sensitive object in an image according to a privacy preference set in an SSI profile list, and add a second DP noise of Laplace distribution to a bounding box surrounding the detected sensitive object to distort and hide the sensitive object.

[0141] The self-sovereign access device (100) can generate a binary mask that distinguishes the area where the second DP noise is added to the PP image into black and white.

[0142] The self-sovereign access device (100) can create a binary mask for the purpose of clearly indicating the location of a sensitive object that is hidden in a PP image.

[0143] The self-sovereign access device (100) can generate a binary mask by making the pixels in the part where the second DP noise is located have a value of 1 (white) and the pixels in other parts have a value of 0 (black).

[0144] The self-sovereign access device (100) uses the generated binary mask only in the subsequent image classification step and does not share it with the outside.

[0145] Figure 4 is a diagram illustrating the process of generating a binary mask.

[0146] In Fig. 4, an example of a PP image and a binary mask can be expressed.

[0147] In Fig. 4, the 'SSI:A profile' sets a human face as a sensitive object, and the 'SSI:B profile' sets the tablet screen as a sensitive object without considering a human face.

[0148] The self-sovereign access device (100) can distort the entire image by performing a perturbation that gradually increases the first DP noise of the differential distribution for the input images (410), but cannot generate a binary mask because the second DP noise of the Laplace distribution is not added.

[0149] On the other hand, the self-sovereign access device (100) can generate a binary mask having pixels in an area where the second DP noise of the Laplace distribution is added to the human face as a sensitive object as a value of 1 (white), regardless of the perturbation that gradually increases the first DP noise of the differential distribution for 'SSI:A Profile' (420).

[0150] In addition, the self-sovereign access device (100) can generate a binary mask having pixels in an area where the second DP noise of the Laplace distribution is added to the screen of the tablet as a sensitive object as a value of 1 (white), regardless of the perturbation that gradually increases the first DP noise of the differential distribution for the 'SSI:B Profile' (430).

[0151] The self-sovereign access device (100) can effectively hide sensitive objects by adding second DP noise to the sensitive objects in the image.

[0152] If there is no sensitive object, the self-sovereign access device (100) can only represent a pure image without the second DP noise added.

[0153] The self-sovereign access device (100) can create a custom image classification model for powerful image classification.

[0154] The generated PP image contains second DP noise, and there may be a possibility of loss of important information during the denoising process.

[0155] The self-sovereign access device (100) can expand the diversity of features by simultaneously utilizing DP images and PP images, and create a custom image classification model that classifies insensitive parts within the image, excluding areas containing second DP noise.

[0156] The custom image classification model can extract common features for DP and PP images and selectively extract the most prominent features from DP and PP images.

[0157] The custom image classification model can adjust attention according to the importance of each channel by utilizing the squeeze and excitation (SE) block as a channel-specific attention mechanism.

[0158] A custom image classification model can be designed to realign attention at each layer for feature maps of DP and PP images, focusing more attention on important channels and ignoring channels of relatively low importance.

[0159] Figure 5 is a diagram explaining the process of extracting features from an image.

[0160] The self-sovereign access device (100) can create a custom image classification model composed of a convolution + SE block.

[0161] The custom image classification model receives a DP image and a PP image as input, operates based on a channel-specific attention mechanism, extracts features from each of the DP image and the PP image, and matrixes these features to create a DP feature map and a PP feature map.

[0162] Custom image classification models can optimize and recalibrate the extracted features by utilizing the functions of the SE block after processing each convolution block.

[0163] A custom image classification model can create a DP feature map and a PP feature map so that features obtained from the DP image and the PP image are complementarily included.

[0164] The self-sovereign access device (100) can combine and integrate DP feature maps and PP feature maps.

[0165] The self-sovereign access device (100) can exclude noisy paths related to DP noise before integrating feature maps.

[0166] To this end, the self-sovereign access device (100) can apply a binary mask to each of the DP feature map and the PP feature map based on Equation 3.

[0167] In particular, the self-sovereign access device (100) can be designed so that the area to which the second DP noise is added in the PP feature map is not considered in the subsequent classification process by specifying the area through a binary mask.

[0168] Equation 3 may be a schematic representation of the generation of masked feature maps from DP images and PP images.

[0169] [Formula 3]

[0170]

[0171] The self-sovereign access device (100) can combine feature maps after applying a binary mask using a weight-based connection method.

[0172] Equation 4 can be a diagram representing weighted coupling.

[0173] [Formula 4]

[0174]

[0175] The self-sovereign access device (100) can combine feature maps according to Equation 4.

[0176] Here, variable α can represent the weight applied to the feature map of the PP image, and β can represent the weight for the feature map of the DP image.

[0177] These weights can be set to satisfy the condition α > β.

[0178] Finally, the self-sovereign access device (100) can deliver the combined feature map, FMc, as various additional layers depending on the specific task or number of classes.

[0179] Figure 6 is a diagram for explaining the process of image classification.

[0180] In step 601, the custom image classification model can receive a PP image as input.

[0181] In step 602, the custom image classification model can extract features from the PP image according to a channel-specific attention mechanism to create a PP feature map.

[0182] In step 603, the self-sovereign access device (100) can apply a binary mask (600) to the PP feature map produced by the custom image classification model.

[0183] In step 611, the custom image classification model can receive a DP image as input.

[0184] In step 612, the custom image classification model can extract features from the DP image according to a channel-specific attention mechanism to create a DP feature map.

[0185] In step 613, the self-sovereign access device (100) can apply a binary mask (600) to the DP feature map produced by the custom image classification model.

[0186] In step 620, the self-sovereign access device (100) can fuse features by weighting and combining the DP feature map and the PP feature map.

[0187] At step 630, the self-sovereign access device (100) can perform layer classification (image classification) on the image using the fused features.

[0188] At step 640, the self-sovereign access device (100) can output the layer classified results.

[0189] The self-sovereign access device (100) can selectively extract the most prominent features of DP images and PP images through a custom image classification model, and can exclude noise patches that obscure sensitive objects in the images from processing by utilizing a binary mask.

[0190] Figure 7 is a diagram illustrating the overall operation of the self-sovereign access device.

[0191] The self-sovereign access device (100) can denoise the first DP noise of the differential distribution in the DP image and restore the original image.

[0192] The self-sovereign access device (100) can recognize sensitive objects within the restored original image according to the privacy preferences set in the SSI profile.

[0193] The self-sovereign access device (100) can generate a PP image by adding a second DP noise of Laplace distribution only to the recognized sensitive object.

[0194] The self-sovereign access device (100) can generate a binary mask that distinguishes an area in a PP image to which second DP noise has been added.

[0195] The self-sovereign access device (100) can extract features from each of the DP image and the PP image through a custom image classification model to create a DP feature map and a PP feature map.

[0196] The self-sovereign access device (100) can apply a binary mask to the DP feature map and the PP feature map to remove the noise path to which DP noise has been added, and then fuse the features by weighting and combining the DP feature map and the PP feature map.

[0197] The self-sovereign access device (100) can classify an image using fused features and output the result.

[0198] Below, FIG. 8 describes in detail the work flow of the self-sovereign access device (100) according to embodiments of the present invention.

[0199] FIG. 8 is a flowchart illustrating a self-sovereign approach for privacy-preserving image generation and robust classification according to one embodiment of the present invention.

[0200] The self-sovereign approach for privacy-preserving image generation and robust classification according to the present embodiment can be performed by a self-sovereign access device (100).

[0201] First, the self-sovereign access device (100) denoises the first DP noise in the DP (Differential Privacy) image to restore it to the original image (810). Step (810) may be a process of removing the first DP noise from a DP image that is perturbed by noise and thus cannot recognize the original, thereby restoring the DP image to the original image.

[0202] A DP image can refer to an image that protects the privacy of data owners by quantitatively modeling privacy.

[0203] The self-sovereign access device (100) can prepare the DP image by perturbing the entire image with the first DP noise of the differential distribution, satisfying Equation 1.

[0204] [Formula 1]

[0205]

[0206] Here, the above △f is a sensitivity of up to 255 of the pixel value defined as the maximum value for a color image, and the above ε and δ may be privacy parameters set by the data owner.

[0207] That is, the self-sovereign access device (100) can prevent the sensitive object from being exposed to the outside by perturbing the image owned by the data owner, which includes the sensitive object, with DP noise having a differential distribution.

[0208] The DP image produced by the self-sovereign access device (100) can cover the entire image, including the sensitive object, with the first DP noise so that the sensitive object is not revealed in the image itself.

[0209] Additionally, the self-sovereign access device (100) recognizes sensitive objects within the restored original image (820), based on the data owner's privacy preferences. Step (820) may be a process of identifying a specific object designated by the data owner as an object to be preferentially hidden in the restored image.

[0210] Here, a sensitive object may be an object that the data owner designates based on his or her subjective judgment, as it is expected that privacy will be seriously violated if exposed externally.

[0211] In the present invention, a sensitive object can be designated based on a Self-Sovereign Identity (SSI) profile.

[0212] The self-sovereign access device (100) can define an object designated by the data owner among multiple objects constituting the SSI (Self-Sovereign Identity) profile as the sensitive object. That is, the self-sovereign access device (100) can determine, in the SSI profile, a specific object designated by the data owner as an object to be hidden according to his / her own priority as a sensitive object.

[0213] Here, an SSI profile can be a data owner profile created in a self-sovereign identity environment. SSI can refer to an identity management model and implementation technology that empowers individuals to control the information they use to prove their identity on websites, services, and applications.

[0214] Additionally, the self-sovereign access device (100) can set the privacy preference, which is differentiated according to the order in which the data is defined, to each of the sensitive objects. That is, the self-sovereign access device (100) can define an object as a sensitive object by granting it a relatively higher privacy preference the earlier the data owner designates the object.

[0215] Accordingly, the self-sovereign access device (100) can sequentially recognize the sensitive object with the highest privacy preference. That is, the self-sovereign access device (100) can sequentially recognize the sensitive object in order of highest privacy preference among multiple objects or sensitive objects in the restored original image.

[0216] Continuing, the self-sovereign access device (100) generates a PP (privacy-preserving) image (830) by adding a second DP noise only to the recognized sensitive object. Step (830) may be a process of generating a PP image in which only the sensitive object is obscured by adding a second DP noise to an area where a specific sensitive object recognized according to a privacy preference is located within the reduced image.

[0217] In generating a PP image, the self-sovereign access device (100) can generate a PP image by adding a second DP noise of a Laplace distribution to a recognized sensitive object.

[0218] The self-sovereign access device (100) can generate the PP image by masking the sensitive object with the second DP noise of the Laplace distribution, satisfying Equation 2.

[0219] [Formula 2]

[0220]

[0221] The second DP noise of the Laplace distribution has a lower ε value than the first DP noise of the differential distribution, which makes the noise estimation and denoising process for the related generated PP image more difficult and allows for more robust protection of sensitive objects from external exposure.

[0222] According to an embodiment, the self-sovereign access device (100) may generate a binary mask for distinguishing an area in the PP image where the second DP noise is masked, along with the generation of the PP image.

[0223] That is, the self-sovereign access device (100) can generate a binary mask that distinguishes the second DP noise by making the pixels in the area where the second DP noise is added in the PP image have a value of 1 (white) and making the pixels in other areas have a value of 0 (black).

[0224] The above binary mask is used in subsequent image classification to identify pixels in the area to which the second DP noise has been added.

[0225] After the PP image is generated, the self-sovereign access device (100) can fuse the features of the DP image and the features of the PP image, and accurately classify the PP image including the second DP image through the fused features.

[0226] That is, the self-sovereign access device (100) can input the DP image and the PP image into a custom image classification model and output a DP feature map and a PP feature map from the custom image classification model.

[0227] The self-sovereign access device (100) can create a custom image classification model that can extract key features of an image by utilizing various known or existing images.

[0228] Additionally, the self-sovereign access device (100) can provide enhanced image classification capabilities through feature fusion and exclusion of areas where second DP noise is added.

[0229] The custom image classification model can extract features from each of the DP image and the PP image input by the self-sovereign access device (100).

[0230] That is, a custom image classification model can learn about DP images to extract DP features, and learn about PP images to extract PP features.

[0231] In addition, the custom image classification model can produce the DP feature map and the PP feature map with the attention readjusted for each layer by adjusting the attention according to the importance of the feature by utilizing the attention mechanism.

[0232] That is, the custom image classification model can adjust and assign attention to each feature by considering the importance of each feature extracted through the squeeze and excitation (SE) block according to the above-mentioned attention mechanism, and can create a DP feature map and a PP feature map by matrixizing multiple features to which each adjusted attention is assigned.

[0233] Here, attention mechanisms can be used to improve the performance of neural networks by allowing the model to focus on the most important input data while generating predictions. This can be accomplished by weighting input data so that the model prioritizes some input attributes over others.

[0234] Thereafter, the self-sovereign access device (100) can apply the binary mask to remove the second DP noise from the DP feature map and the PP feature map, and then fuse the features by weighted combining the DP feature map and the PP feature map. That is, the self-sovereign access device (100) can apply the binary mask to each of the DP feature map and the PP feature map to distinguish the second DP noise from these feature maps, and combine only the remaining portions of the feature maps in which the distinguished second DP noise is not involved, thereby fusion of the features.

[0235] Weighted combining can mean combining feature maps in a state where a relatively large weight is given to a feature map that receives more attention depending on the implementation environment, through the preceding attention mechanism.

[0236] In addition, the self-sovereign access device (100) can classify and identify the PP image using the fused features. That is, the self-sovereign access device (100) can classify the image by layer classifying the fused features, thereby identifying what the generated PP image is, and output the identified result to the outside.

[0237] According to one embodiment of the present invention, a self-sovereign approach and device for privacy-preserving image generation and robust classification can be provided, which efficiently generates privacy-preserving images that precisely hide only specific sensitive objects without distorting the entire image.

[0238] Additionally, according to one embodiment of the present invention, it is possible to enable a data owner to designate specific and important sensitive information that he or she wishes to protect.

[0239] In addition, according to one embodiment of the present invention, even if DP noise is included in a wide area, an image can be accurately classified.

[0240] In addition, according to one embodiment of the present invention, since sensitive objects are excluded from the learning and prediction process, the resistance of a model to model inversion attacks aimed at reconstructing sensitive objects of an image can be strengthened.

[0241] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0242] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0243] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0244] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a DP (Differential Privacy) image, a step of denoising the first DP noise and restoring the original image; A step of recognizing a sensitive object within the restored original image according to the privacy preference of the data owner; and A step of generating a PP (privacy-preserving) image by adding a second DP noise only to the above-mentioned recognized sensitive objects. A self-sovereign approach for privacy-preserving image generation and robust classification, including 2. In paragraph 1, A step of defining an object designated by the data owner among multiple objects constituting an SSI (Self-Sovereign Identity) profile as the sensitive object, and setting the privacy preference, which is differentiated according to the order of definition, to each of the sensitive objects. Including more, The step of recognizing the above sensitive object is: Step of sequentially recognizing the sensitive object with the highest privacy preference A self-sovereign approach for privacy-preserving image generation and robust classification, including 3. In paragraph 1, A step of preparing the DP image by perturbing the entire image with the first DP noise of the differential distribution so as to satisfy Equation 1. Including more, The above formula 1 is, person, - The above △f is the sensitivity of up to 255 of the pixel value defined as the maximum value for the color image, and the above ε and δ are privacy parameters set by the data owner. A self-sovereign approach for privacy-preserving image generation and robust classification.

4. In paragraph 1, The steps for generating the above PP image are: A step of generating the PP image by masking the sensitive object with the second DP noise of the Laplace distribution, satisfying Equation 2. Including, The above formula 2 is, person, A self-sovereign approach for privacy-preserving image generation and robust classification.

5. In paragraph 1, In the above PP image, a step of generating a binary mask that distinguishes the second DP noise by making pixels in the area where the second DP noise is added have a value of 1 (white) and pixels in other areas have a value of 0 (black). A self-sovereign approach for privacy-preserving image generation and robust classification, including:

6. In paragraph 5, A step of inputting the above DP image and the above PP image into a custom image classification model and outputting a DP feature map and a PP feature map from the custom image classification model. Including more, The above custom image classification model is, Extract features from each of the above DP image and the above PP image, By utilizing the attention mechanism, attention is adjusted according to the importance of the feature, thereby producing the DP feature map and the PP feature map with the attention readjusted for each layer. A self-sovereign approach for privacy-preserving image generation and robust classification.

7. In paragraph 6, A step of applying the above binary mask to remove the second DP noise from the DP feature map and the PP feature map, and then merging the features by weighted combining the DP feature map and the PP feature map; and A step of classifying and identifying the PP image using the above fused features. A self-sovereign approach for privacy-preserving image generation and robust classification, including 8. A privacy-preserving image generation module that denoises the first DP noise in a DP (Differential Privacy) image to restore the original image, recognizes a sensitive object in the restored original image according to the privacy preference of the data owner, and creates a PP (privacy-preserving) image by adding the second DP noise only to the recognized sensitive object. A self-sovereign approach for privacy-preserving image generation and robust classification, including:

9. In paragraph 8, A preprocessing module that defines an object designated by the data owner among multiple objects that constitute an SSI (Self-Sovereign Identity) profile as the sensitive object, and sets the privacy preference, which is differentiated according to the order of definition, to each of the sensitive objects. Including more, The above privacy-preserving image generation module, Sequentially recognizing the sensitive objects with the highest privacy preference, A self-sovereign approach for privacy-preserving image generation and robust classification.

10. In paragraph 8, A preprocessing module that prepares the DP image by perturbing the entire image with the first DP noise of the differential distribution satisfying Equation 1. Including more, The above formula 1 is, person, - The above △f is the sensitivity of up to 255 of the pixel value defined as the maximum value for the color image, and the above ε and δ are privacy parameters set by the data owner. A self-sovereign approach for privacy-preserving image generation and robust classification.

11. In paragraph 8, The above privacy-preserving image generation module, By satisfying Equation 2, the PP image is generated by masking the sensitive object with the second DP noise of the Laplace distribution, The above formula 2 is, person, A self-sovereign approach for privacy-preserving image generation and robust classification.

12. In paragraph 8, The above privacy-preserving image generation module, In the above PP image, a binary mask is generated to distinguish the second DP noise by making pixels in the area where the second DP noise is added have a value of 1 (white) and pixels in other areas have a value of 0 (black). A self-sovereign approach for privacy-preserving image generation and robust classification.

13. In paragraph 12, The above self-sovereign access device is, A classification module that inputs the DP image and the PP image into a custom image classification model and outputs a DP feature map and a PP feature map from the custom image classification model. Including more, The above custom image classification model is, Extract features from each of the above DP image and the above PP image, By utilizing the attention mechanism, attention is adjusted according to the importance of the feature, thereby producing the DP feature map and the PP feature map with the attention readjusted for each layer. A self-sovereign approach for privacy-preserving image generation and robust classification.

14. In paragraph 13, The above classification module, By applying the above binary mask, the second DP noise is removed from the DP feature map and the PP feature map, and then the DP feature map and the PP feature map are combined by weighted combination to fuse the features. Using the above fused features, the PP image is identified by classifying it as an image. A self-sovereign approach for privacy-preserving image generation and robust classification.

15. A computer-readable recording medium recording a program for executing the method of paragraph 1.

Images