De-identification method, de-identification-based bio-information processing method, and device thereof

Abstract

The present disclosure relates to a de-identification method, a bio-information processing method based on de-identification, and a device thereof. It relates to a technology for protecting visual features within an image, and a de-identification method receives an image including a face region, extracts feature information about the face region from the input image, transforms the extracted feature information to provide guidance for a part to be generated within the face region, and performs de-identification based on the provided guidance.

Description

Technical Field

[0001] This disclosure relates to a technique for protecting visual features within an image, and more specifically, to a method for de-identifying visual features within an image, a method for processing biometric information based on de-identification, and an apparatus using the method. Background Technology

[0002] With the advancement of artificial intelligence technology, various forms of personal information are constantly emerging. Among them, in addition to facial information, image information representing various characteristics of an individual is also used as personal information.

[0003] Visual features are used to distinguish multiple objects from one another. Visual features can be represented by various properties of visual objects in visual data. For recognizing objects within an image, various image features can be used, such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), Speed-Up Robust Feature Transform (SURF), Haar, Ferns, Local Binary Pattern (LBP), and Modified Census Transform (MCT). These visual features not only represent the unique characteristics of objects but sometimes also exhibit the unique characteristics of specific individuals, making them crucial for recognition and requiring preservation.

[0004] With the advent of the artificial intelligence era, facial recognition technology is becoming increasingly widespread. Facial recognition involves extracting visual features from images, including faces, to distinguish them, which can raise privacy concerns. For example, visual features (i.e., personal information, which can identify an individual from facial information of a person moving on the street or from photos shared on social networks) can be leaked.

[0005] To prevent such leakage of personal information, various de-identification techniques have been proposed, which blur visual features corresponding to personal information. The patent document presented below proposes a method for detecting and monitoring facial regions within a video and applying mosaic effects to these regions to protect personal privacy. However, while applying mosaic effects to the entire facial area can protect personal information, it constitutes a completely anonymized technique that makes it impossible to identify an individual, and therefore is not suitable for situations where visual identification of the person in the video is still required.

[0006] Therefore, there is a need to develop new technologies that can reveal a certain level of visual features within an image while also protecting personal information.

[0007] [Patent Document] Korean Patent Registration No. 10-1612735, "Selective Region Mosaicization Method Based on Facial Analysis". Summary of the Invention

[0008] Various embodiments of this disclosure aim to address the problem of personal information leakage that may occur when there are insufficient protection measures for visual features in images used in image recognition and the like. They overcome the weakness that traditional de-identification techniques significantly modify or remove visual features in the original image, making it difficult to identify the original person. They also overcome the limitation that when de-identification is performed irreversibly, the visual features in the original image cannot be recovered if identification data is needed later.

[0009] To address the aforementioned technical problems, in one aspect, a de-identification method is provided, the de-identification method comprising: receiving an image including a facial region; extracting feature information about the facial region from the input image; transforming the extracted feature information to provide guidance for generating a portion within the facial region; and performing de-identification based on the provided guidance.

[0010] The de-identification method may also include performing at least one of the following preprocessing steps on the input image: shadow removal, noise removal, image modification, and color correction.

[0011] Feature extraction may include at least one of the following: extracting region-specific features of different region sizes from the input image; and performing masking processing on the input image with grid cells of different sizes.

[0012] Region-specific feature extraction may include: segmenting the entire image region into dedicated regions of different sizes; for each segmented dedicated region, extracting features of the corresponding image by transforming from a high-dimensional image to a low-dimensional image, and transforming the extracted features into a latent space; and during the transformation process of image data corresponding to a relatively large region, providing image information corresponding to a relatively small region as input to each layer, wherein the input is adjusted to match the size of the larger region.

[0013] Performing masking processing may include: setting at least two or more segmentation criteria; segmenting the entire image region into a grid of predetermined size according to the set segmentation criteria, extracting structural features of the corresponding image for each grid, wherein grids segmented according to different segmentation criteria overlap in some regions; and aggregating the features extracted from each grid for each region to construct structural features corresponding to the entire image region. Performing masking processing may include extracting skin features from relatively smaller grids in the segmented grid.

[0014] Providing guidance may include removing or modifying feature information corresponding to the newly generated portion of the facial region according to preset transformation rules. The de-identification method may also include pre-storing or sending at least one or a combination of the feature information and transformation rules to a device configured to re-identify the de-identified facial image.

[0015] Performing de-identification can include using generative artificial intelligence and generating de-identified facial images based on guidance provided corresponding to the parts to be newly generated.

[0016] Feature extraction may include learning parameters associated with the forward transformation from a facial image to noise using a diffusion model. Deidentification may include generating a deidentified facial image from the noise through an inverse transformation of the diffusion model, guided by the removal or modification of feature information. Deidentification may include generating the facial image during the inverse transformation of the diffusion model by first using features extracted from the image in a relatively large grid region, and then using features extracted from the image in progressively smaller grid regions.

[0017] To address the aforementioned technical problems, in another aspect, a method for processing biometric information is provided. The method includes the following steps: receiving a de-identified image; receiving feature information from an original image and at least one or a combination of transformation rules for de-identification; and performing re-identification using the received feature information from the original image or the received transformation rules to recover a facial image from the input de-identified image. The de-identified image is generated by: extracting feature information about a facial region from the original image; transforming the extracted feature information to provide guidance for generating a portion within the facial region; and performing de-identification based on the provided guidance.

[0018] Performing de-identification may include using the feature information of the original image to regenerate feature information that has been removed or modified according to a set transformation rule to restore the facial image.

[0019] De-identified images can be generated in the following ways: for each specific region of different sizes from the input image, features of the corresponding image are extracted by transforming from a high-dimensional image to a low-dimensional image, and the extracted features are transformed into a latent space to extract region-specific features; or a masking process is performed to extract structural features of the corresponding image from each grid of different sizes of the input image, and the features extracted from each grid are aggregated for each region to extract structured feature information.

[0020] Furthermore, in the following description, the present invention provides a computer-readable recording medium in which programs for performing de-identification methods and biometric information processing methods have been recorded on a computer.

[0021] According to one embodiment of this disclosure, a de-identification device is provided, the de-identification device including a memory and a processor, the memory being configured to store a program for receiving an image including a facial region and de-identifying the received image, the processor being configured to execute the program stored in the memory, wherein the program includes the following instructions: extracting feature information about the facial region from the image including the facial region; transforming the extracted feature information to provide guidance for a portion to be generated within the facial region; and performing de-identification based on the provided guidance.

[0022] According to another embodiment of this disclosure, a biometric information processing device is provided, the biometric information processing device including a memory and a processor, the memory being configured to store a program for receiving a de-identified image and re-identifying it, the processor being configured to execute the program stored in the memory, wherein the program includes the following instructions: receiving a de-identified image; receiving feature information of an original image and at least one or a combination of a set of transformation rules for de-identification; and performing re-identification using the received feature information of the original image or the received transformation rules to recover a facial image from the input de-identified image, and the de-identified image is generated by: extracting feature information about a facial region from the original image; transforming the extracted feature information to provide guidance for generating a portion within the facial region; and performing de-identification based on the provided guidance.

[0023] According to various embodiments of this disclosure, de-identification can be performed by extracting region-specific features or transforming feature information obtained through grid-based masking processing. This can protect visual features within an image and retain partial information reflecting features from each region, thereby generating a natural face that is easily recognizable to humans. Furthermore, re-identification can be achieved by separately storing and using feature information or transformation rules to restore the visual features of the original image. Attached Figure Description

[0024] The accompanying drawings are included to provide a further understanding of the present disclosure and form part of the detailed description. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the technical features of the present disclosure.

[0025] Figure 1 It is a view used to describe the process of de-identifying or re-identifying visual features within an image, including personal information.

[0026] Figure 2 This is a view showing the results of applying various de-identification algorithms to faces.

[0027] Figure 3This is a view illustrating an overview of the de-identification process according to various embodiments of this disclosure.

[0028] Figure 4 This is a flowchart illustrating a de-identification method according to one embodiment of the present disclosure.

[0029] Figure 5 This diagram illustrates the process of extracting features on a region-by-region basis when extracting feature information about facial regions.

[0030] Figure 6 It is a view used to describe the mesh-based masking process performed when extracting feature information about facial regions.

[0031] Figure 7 It is a view used to describe the process of generating deidentified images using a diffusion model.

[0032] Figure 8 This is a flowchart illustrating a method for processing biometric information based on de-identification, according to another embodiment of the present disclosure.

[0033] Figure 9 This is a block diagram illustrating a de-identification device and a re-identification device according to yet another embodiment of the present disclosure. Detailed Implementation

[0034] Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Detailed descriptions of known technologies will be omitted if they might mislead the gist of embodiments of the present disclosure. Furthermore, throughout this disclosure, unless otherwise stated, “including” a component means that other components may be further included, rather than excluding other components.

[0035] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. Singular expressions include the meaning of plural forms unless they have a clear meaning in the context. In this disclosure, expressions such as “comprising” or “having” are intended to indicate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and should not be construed as precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0036] Unless otherwise stated, all terms used herein, including technical or scientific terms, have the same meaning as those commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries may be understood to have the same meaning as those used in the context of the relevant field and should not be construed as having an idealized or overly formal meaning unless they are clearly specified in this disclosure.

[0037] Figure 1 It is a view used to describe the process of de-identifying or re-identifying visual features within an image, including personal information.

[0038] When the original image 110 is input and a series of image processing procedures 120, 130 and 140 are performed on the original image 110, a de-identified image 150 can be obtained. Here, the target of de-identification can be a "face" that concentrates personal information or visual features that can identify an individual.

[0039] A series of image processing steps include setting a region of interest (ROI) and replacing the ROI with another image or data. When it is necessary to recover de-identified feature information in the future, the original image can be recovered by a legally authorized person. Technically, the likelihood of recovery varies depending on the operating strategies of the owner or administrator, and... Figure 1 Assuming recovery is possible, that is, a reversible de-identification process.

[0040] The image processing procedure for de-identification can include the following three operations.

[0041] 1) Image preprocessing operation 120: Before image processing, the system adjusts the size of the original image or reduces noise in the original image. For example... Figure 1 As shown, in images that include faces, the system can detect people first by prioritizing the background or other objects.

[0042] 2) Object and Region Extraction Operation 130: The system extracts the image region of the target object based on the de-identified target. In particular, objects can be aligned around the ROI, or specific objects can be extracted.

[0043] 3) Visual de-identification operation 140: Specific de-identification mechanisms can be applied, such as black box processing, blurring processing, pixelation processing, eigenface, etc., to remove or modify the unique features of the corresponding person.

[0044] Visual de-identification operations 140 can be implemented in various forms.

[0045] One approach is to use an irreversible loss function to modify identity data. This method cannot recover the original information from the transformation, but it reduces the probability of re-identification in terms of de-identification. For visual data, this is a typical de-identification process.

[0046] Another approach involves encryption assistance, enabling re-identification when necessary. In this case, since the information content is not reduced, re-identification can be attempted if needed, but there is an inherent risk of personal information leakage.

[0047] Another approach involves de-identification by removing or altering a person's facial features. As a result, a "me who is not me" may be created. This method removes identifying features from the face, and while other remaining features in the image still give the appearance of the face, the de-identified face makes it difficult to clearly identify the individual. In some cases, this method may produce the impression of a person similar to the original image, subtly making the person look different. Furthermore, acquaintances of the person may perceive the image as similar, but a facial recognition algorithm may determine that the two people are different. This is because unique facial features are removed or modified, causing the facial recognition algorithm to identify the corresponding person as a different individual. In various embodiments of this disclosure, protected privacy refers to an individual's facial features, but these features refer to the means of distinguishing an individual's identity from the perspective of a device, machine, or algorithm attempting to identify a person from the perspective of others.

[0048] refer to Figure 1 Visual deidentification operation 140 illustrates that deidentification can be applied to a previously acquired facial image to obtain a deidentified facial image. In this case, unique information determining the transformation rules, feature information, or encryption method used for the deidentification process can be stored separately in preparation for future recovery requests. Figure 1 The encryption key 145 is shown, but other de-identification techniques can be applied in addition to encryption. Next, when a request to recover the de-identified facial image occurs, re-identification can be performed using the pre-stored encryption key 145, transformation rules, or feature information to obtain the recovered facial image.

[0049] Figure 2 The results of applying various de-identification algorithms to faces are shown, and techniques for removing identifiable visual features are illustrated.

[0050] Figure 2 (a) shows the original facial image to be deidentified.

[0051] Figure 2 (b) shows the de-identification results using the black-box technique. This technique differs slightly from typical mosaic techniques because a portion of the image is erased. While mosaic techniques tile a portion, black-box techniques erase the entire area, resulting in superior de-identification performance; however, this approach may also lead to significant information loss.

[0052] Figure 2 (c) shows the de-identification result using obfuscation techniques. This technique prevents information leakage by splitting the data and reducing its clarity. However, obfuscation is not recommended because recovery techniques improve rapidly with each level, and its operational performance is similar to mosaic.

[0053] Figure 2 (d) illustrates the de-identification result using pixelation techniques. This technique encompasses all methods of removing or transforming data containing personal information, and blurring corresponds broadly to this category. However, since the purpose is to modify specific information or image areas, this technique can be used for restoration or other purposes if needed.

[0054] Figure 2 (e) illustrates the de-identification results using the Eigenface technique. This technique processes facial information while preserving some personal attribute information, and the Eigenface method transforms the feature information from an individual's facial image. By transforming unique information in this way, one person can be transformed into another using a small amount of data, and can also be restored using a small amount of data if needed. However, a key challenge of this technique is how to ensure the security of personal attribute information.

[0055] Figure 2 (f) shows the de-identification result using the K-same technique. In this technique, faces are represented using overlapping facial information from multiple individuals, and by overlapping facial information from multiple people, specificity and limitation on a single individual can be prevented. The K-same technique is relatively easy to implement, but it can use frequency characteristics to distinguish multiple faces and is not considered a complete de-identification method.

[0056] Figure 2 (g) illustrates the de-identification result using a cartoonization technique. This technique is a style transfer that combines or modifies data included in existing facial data into different formats. The cartoonization technique shown guarantees a degree of anonymity while preserving the characteristics of the original photograph. However, even in this case, a significant portion of the data included in the original photograph is lost, making recovery extremely difficult or requiring separate storage techniques. The advantage of style transfer techniques is that they preserve some of the original characteristics, and the modified photograph also retains the original characteristics, resulting in a more natural appearance.

[0057] The various implementation methods presented below are based on generative de-identification techniques, which are proposed to address the weaknesses of various de-identification technologies. Unlike traditional mathematical models or redundant data, generative de-identification techniques use data from a cognitive perspective. Therefore, unlike traditional techniques, generative de-identification techniques do not require user resistance and can be restored or transformed using a small amount of data as needed.

[0058] Various embodiments of this disclosure propose a guided generation technique that preserves shape and characteristics. The guided generation technique identifies feature points of facial information and uses these feature points as guides to generate facial information. That is, in the process of generating de-identified virtual facial data, this technique preserves specific portions reflecting features from each region, rather than feature points, and uses the guides to generate other portions. For this purpose, guides are provided for the portions to be masked, and a de-identified facial image is generated based on these guides. In particular, this technique is used to generate facial features that prevent the loss of data to be protected by adding additional data to the facial features. The addition of additional data can be achieved through the following synthesis method.

[0059] 1) Facial Feature and Segmentation Synthesis: This method identifies facial features, segments features, and generates facial features. It modifies key facial features while preserving existing data as much as possible, even when modifying only the desired portion in cases of low integrity or heterogeneity. Furthermore, the segmentation itself provides security.

[0060] 2) Facial Feature and Region Synthesis: Previous segmentation methods cropped and processed feature parts, while this method pre-segmented multiple parts of the face and transformed the classification data. This method reveals the modification of features in areas that people perceive as cognitively important. Therefore, this method addresses general aspects rather than the importance of the data.

[0061] 3) Facial feature, facial region, and mesh-based masking synthesis: In addition to using features for each region, this method also uses a mesh to mask each part. Through masking, local data is preserved and then expanded, thereby achieving de-identification.

[0062] Figure 3 This is a view illustrating an overview of the de-identification process according to various embodiments of the present disclosure. During the de-identification process, decisions can be made regarding which areas to target, what information to modify, and how to modify the feature information.

[0063] First, an image including the facial region is acquired (310) and preprocessed (320), and then the target region (e.g., the face) is extracted (330). Various embodiments of this disclosure are for images that have undergone significant preprocessing (e.g., shadow removal, noise removal, etc.), and it is assumed that distortions or color changes in the acquired image can be corrected through the preprocessing process. Therefore, it is assumed that the image is acquired and preprocessed to achieve an appropriate level of image quality, allowing for partial occlusion.

[0064] Various embodiments of this disclosure target the face, but may include facial information from similar animals and insects. That is, unlike traditional facial recognition technologies that limit key feature points to the face, this technology incorporates individual facial features. Traditional facial recognition relies on the use of predetermined key feature points for speed, but this disclosure may include all the information needed to identify a specific object.

[0065] Then, features can be extracted from the extracted target region using at least one of the two methods.

[0066] First, during the region-based feature extraction process 340, features are extracted by modifying the input process of size-specific segments from the target image used for learning. To extract features, the features extracted from the size-specific segments are used as intermediate information input into the methods of each layer during the process of transforming information from each image region into the latent space.

[0067] Secondly, during the grid-based structural feature extraction process 350, the facial information is divided into grids of various sizes, features are extracted, and the feature sets of each grid region are overlapped to generate structured feature information.

[0068] Then, the features to be maintained are retained, and the features to be removed or modified are transformed according to preset rules or strategies (360). Thus, a guided image is generated (370), thereby achieving image information de-identification (380).

[0069] Figure 4 This is a flowchart illustrating a deidentification method according to one embodiment of the present disclosure, and describes a series of image processing procedures that can be performed by a deidentification device including at least one processor.

[0070] In operation S410, the de-identification device receives an image including a facial region. In this case, at least one of the following preprocessing techniques can be performed on the input image: shadow removal, noise removal, image modification, and color correction.

[0071] In operation S430, the de-identification device extracts feature information about facial regions from the image input via operation S410. This process may include at least one of the following: extracting region-specific features of different region sizes from the input image; and performing masking on the input image using grid cells of different sizes.

[0072] Feature extraction can be performed by extracting features from the entire face or by finding predefined features. However, these methods may have a side effect: information from regions with strong features can obscure information from other regions. To prevent this, various embodiments of this disclosure propose techniques for extracting features from a variety of regions.

[0073] Figure 5 This is a view (340) used to describe the process of extracting features on a region-by-region basis when extracting feature information about facial regions.

[0074] When extracting features from an entire image region, the structure of the learning system is determined regardless of the size of the input image; therefore, feature extraction is performed by transforming the image to a predetermined size. Consequently, if the image may not include features larger than or equal to a certain size, there is a risk of losing important information. Therefore, one embodiment of this disclosure proposes a method for extracting features by inputting information about each image region of each size into individual layers during the process of transforming information from the image region to the latent space. (Reference) Figure 5 It can be seen that during the process of extracting features of the largest region (341), information about the image regions extracted in units of different small sizes is added to each layer. Since there is a difference between the information input during feature point learning and the information directly input from each layer, Figure 5 This demonstrates a technique that operates differently depending on the size of the target image input for learning. That is, by extracting features for different input processes targeting large and small regions, the final latent vector 342 can be obtained. Regarding the resulting parameters, Figure 5 The implementation can further highlight features that are significant in small regions. For example, in the case of small dots on a face, it is difficult to reflect the features of the small dots by extracting features from the entire image region, but in the method of extracting features on a region-by-region basis, the information representing the small dots is processed separately, allowing the visual features that accurately reflect the corresponding information even at small sizes.

[0075] Regarding the process of transforming information from image regions to latent space. Figure 5 A method is shown for adding information about each image region extracted from a relatively small image region (e.g., 256 × 256, 128 × 128, 64 × 64, etc.) relative to the largest image region (512 × 512) to the respective layers, but this is only one implementation and is not limited thereto. For example, information about the image regions extracted from each region can be accumulated in a cascaded manner. That is, by adding information extracted from a 64 × 64 image region to a 128 × 128 image region, information extracted from a 128 × 128 image region to a 256 × 256 image region, and information extracted from a 256 × 256 image region to a 512 × 512 image region, information obtained from image regions of different sizes can be layered and input into the next stage of the latent space transform processing.

[0076] In summary, the process of extracting features from each region can segment the entire image region into exclusive regions of different sizes. For each segmented exclusive region, features of the image can be extracted by transforming the high-dimensional image into a low-dimensional image and then mapping it to the latent space. During the transformation process of the image data corresponding to the relatively large region, image information obtained from the relatively small region can be provided as input to each layer, where the input is adjusted to match the size of the larger region.

[0077] The aforementioned techniques for extracting features from each region highlight small features. The technique to be introduced next segments facial information into grids of different sizes, processes the segmented facial information, and then reassembles the processed facial information to generate structured feature information.

[0078] Figure 6 It is a view used to describe the mesh-based masking (350) process performed when extracting feature information about facial regions.

[0079] refer to Figure 6 For example, a method could be applied as follows: Divide the entire 512 × 512 image into a grid consisting of one image, four quarter-sized (256 × 256) images, nine ninth-sized (171 × 171) images, sixteen eleventh-sized (128 × 128) images, and twenty-five twentieth-sized (82 × 82) images; process each grid region; and then reconstruct the processed region. (See reference.) Figure 6 The diagram shows feature information 351, 352, 353, and 354 derived from grids of different sizes, illustrating how these are aggregated into a structured feature set.

[0080] The grid structure is not configured to repeatedly divide the original region into progressively smaller sizes, such as 1→1 / 4→1 / 16, but rather to divide it according to various ratios, such as 1, 1 / 4, 1 / 9, and 1 / 16, including some overlapping regions. That is, grid regions divided into different sizes can derive features from different perspectives relative to adjacent regions. In this embodiment, information about skin color in the multiple segments and segmented regions can be included to extract small feature points. Regarding the overlapping grid regions, four 300 × 300 images can also be constructed instead of... Figure 6 The image shown consists of four 256 × 256 images. In this case, considering the original image size of 512 × 512, the four segmented grids can be implemented as overlapping each other.

[0081] Feature information 351, 352, 353, and 354, derived from grids of different sizes, are stacked to correspond to the entire image region, thus forming a structured feature set. In this case, the entire structured feature set includes features extracted from various perspectives (grid segmentation methods) across the entire image region. Therefore, depending on the transformation rules or purpose, the structured feature set can be used in whole or in part. Figure 6 The partial feature set 357 shown includes feature information derived from grids of different sizes accumulated in the corresponding image regions, and includes, for example, feature information about four 128 × 128 images. To focus on features in specific regions rather than the entire image region, only a portion of the structured feature set corresponding to the corresponding region can be used. For example, to focus on individual features related to the lips in an image that includes the entire body or face of a person, only a portion of the structured feature set corresponding to the lip region can be extracted and used for object recognition, transformation, anonymization, etc.

[0082] Since feature point-centric extraction techniques for the entire region cannot reflect the overall characteristics of that region, this embodiment divides the entire region into grids of various preset sizes, and constructs characteristic information based on these grids, thereby reflecting information about skin features such as spots, freckles, skin color, and skin age. Therefore, feature information is obtained from a relatively large image region, and features of the skin region are extracted and processed into information from smaller grid segments.

[0083] In summary, the masking process can set at least two or more segmentation criteria, dividing the entire image region into grids of predetermined size according to the set segmentation criteria, so as to extract the structural features of the corresponding image for each grid. Grids segmented by different segmentation criteria can overlap in some regions, and for each region, the features extracted from each grid are aggregated into structural features corresponding to the entire image region. In particular, during the masking process, skin features can be extracted from the relatively smaller grids in the segmented grids.

[0084] Return to reference Figure 4 In operation S450, the de-identification device transforms the feature information extracted in operation S430 to provide guidance for the portion to be generated within the facial region. In this case, feature information corresponding to the newly generated portion within the facial region can be removed or modified according to preset transformation rules.

[0085] Previously, in operation S430, region-based feature extraction, grid-based feature extraction, and structuring were proposed to obtain individual features, and in operation S450, these were used to remove or modify information. The removal or modification of information can be determined according to the user's strategy or rules, and the process of generating an image based on the modified information corresponds to the next operation, namely, the guided image generation process (operation S470). These transformation rules can be used to set the type of manipulation to be applied to the object set as an indicator of individual features. For example, it can indicate the modification of heatmaps with a specific intensity or higher. A heatmap is a tool that visually represents the degree to which data is concentrated in a specific segment and can be used to visualize the features of image data. Therefore, the de-identification method of this disclosure can use heatmaps to identify which features within a facial image have been focused, and compare the intensity within the heatmap with a threshold to reduce the intensity value in regions with an intensity greater than or equal to the threshold, thereby attenuating individual features. Furthermore, when individual features have been pre-generated as a structured feature set, the desired feature transformation can be achieved by manipulating the corresponding feature set as a whole or in part.

[0086] From an implementation perspective, features can be removed or modified by applying transformation methods that utilize a typical human feature set. In many cases, features are transformed based on similar body types or ethnicities, and such modified information removes individual characteristics, thus presenting a person commonly seen in the surrounding environment. When the transformation methods and rules are stored in this process, they may be used to recover the feature information of the image in the future. Therefore, when recovery is needed, it is preferable to record or share the transformation methods, strategies, or rules, and even the feature information, separately. In this case, when storing a structured feature set, the feature set can be stored in whole or in part. However, considering the storage space required to store a structured feature set, only a portion of the feature set can be recorded or shared. For example, in the case of closed-circuit television (CCTV), deidentified images may need to be reidentified for public purposes. Therefore, at least one or a combination of the transformation methods, transformation strategies or rules, and feature information can be stored in the storage device of the commissioning agency or a separate trusted agency, and authorized users (e.g., police officers and officials) can use this information to recover the original image from the deidentified image.

[0087] In summary, by pre-storing or sending at least one of the feature information and transformation rules, or a combination thereof, to a device for re-identifying deidentified facial images, the deidentified facial images can be used for subsequent image restoration.

[0088] Alternatively, storing the original image separately for recovery could be considered; however, due to the nature of images, this method requires storing a large amount of data and carries the security vulnerability of immediately exposing the original image in the event of a data leak. In contrast, the transformation rule or minimal information retention method proposed in this embodiment can keep storage space at a very small level, and because it requires a technical understanding of the recovery process, it offers the advantage of being relatively unaffected by typical information leakage threats.

[0089] In operation S470, the de-identification device performs de-identification based on provided guidance. This process can utilize generative artificial intelligence (AI) and generates a de-identified facial image based on guidance provided corresponding to the newly generated portion.

[0090] Guided image generation techniques can begin by using features extracted from a large image region, followed by progressively smaller features. Furthermore, structured feature information extracted from a grid region is used to generate the image. This process allows for the combined use of features derived from various methods. Additionally, when shape transformations within the image are required, features extracted from the large image region can be modified, and when variations such as skin color are needed, information extracted from the grid can be used to achieve feature transformations.

[0091] From an implementation perspective, various generative AI algorithms and models can be used, and application techniques based on diffusion models will be provided below.

[0092] Figure 7 This is a view describing the process of generating de-identified images using a diffusion model. A diffusion model is a generative AI technique that uses a forward process (or diffusion process) to transform data into complete noise while gradually adding noise to the data; conversely, it uses a reverse process to generate data through a denoising process that gradually recovers the data from the noise. A detailed description of the diffusion model will be omitted.

[0093] refer to Figure 7 In the process of extracting feature information, parameters for the positive transformation from facial images to noise are learned through a diffusion model (345). This learning process can be, for example, by learning from... Figure 5 The process of extracting features from each region in the process or Figure 6 This is achieved through a grid-based masking process.

[0094] Then, during the de-identification process, guided by the removal or modification of feature information (360), a de-identified facial image is generated from the noise via the inverse transform of the diffusion model (370). In this case, during the inverse transform of the diffusion model, features extracted from the image in a relatively large grid region can be used first, and features extracted from the image in progressively smaller grid regions can be used to generate the facial image. (Refer to...) Figure 7 It can be seen that, targeting Figure 6 The feature information 351, 352, 353, and 354 extracted from each grid size are provided to the inverse transform operation in descending order of size. Through this generation process 370, a de-identified image that meets the requirements of feature transform 360 is finally generated.

[0095] like Figure 7 As shown, the diffusion model proposed in this embodiment can learn an image (345) by obtaining features extracted from each region during the forward transformation or by extracting features through mesh-based masking. Furthermore, by gradually shrinking the target region during the inverse transformation while inputting these acquired features, a de-identified image satisfying the feature transformation (360) requirement can be generated (370) without losing the remaining features of the facial region. Specifically, during the image generation process 370, features extracted through the learning process 345 are progressively applied for each region size.

[0096] The diffusion model proposed in this embodiment can be implemented using stable diffusion or its derivative models. Therefore, Figure 7 The diffusion model can include components in the latent space, such as U-Net, autoencoder, transformer, and attention mechanism, and handles noise-to-image modification and inverse transform mapping through the diffusion function and U-Net.

[0097] The guided de-identification method has been described above. The following section will propose a method to recover the original facial image through re-identification.

[0098] Figure 8 This is a flowchart illustrating a method for processing biometric information based on de-identification according to another embodiment of the present disclosure, and describes a series of image processing procedures that can be performed by a biometric information processing device (or re-identification device) including at least one processor.

[0099] In operation S810, the biometric information processing device receives a de-identified image. Here, feature information about the facial region is extracted from the original image, the extracted feature information is transformed to provide guidance for the portion to be generated within the facial region, and de-identification is performed based on the provided guidance to generate the de-identified image.

[0100] Furthermore, de-identified images can be generated by extracting features from each dedicated region of different sizes from the input image through a transformation from a high-dimensional image to a low-dimensional image, and then transforming the extracted features into the latent space to extract region-specific features. Alternatively, de-identified images can be generated by performing masking to extract structural features from each grid of different sizes in the input image, and then aggregating the features extracted from each grid for each region to extract structured feature information.

[0101] In operation S830, the biometric information processing device receives feature information of the original image and at least one or a combination of transformation rules in a set for de-identification.

[0102] In operation S850, the biometric information processing device performs re-identification using the feature information or transformation rules of the original image received via operation S830 to recover a facial image from the deidentified image input via operation S810. More specifically, the re-identification process can recover the facial image by using the feature information of the original image to regenerate feature information that has been removed or modified according to the set transformation rules. In particular, considering that a structured feature set can be used in whole or in part during image deidentification processing, it is worth noting that it is possible to recover the facial image using only a portion of the modified feature information. Considering the available data storage space required for re-identification, only a portion of this feature set can be used for deidentification or re-identification.

[0103] Figure 9 This illustrates a hardware configuration reconfiguration according to yet another embodiment of the present disclosure. Figure 4 and Figure 8 The block diagrams of the de-identification device 10 and the re-identification device 20 in the image processing procedure are shown below. Therefore, to avoid overlapping descriptions, this document briefly describes the components of each device and focuses on their functions.

[0104] The de-identification device 10 includes a memory 13 and a processor 12. The memory 13 stores a program for receiving an image including a facial region and de-identifying the received image, and the processor 12 executes the program stored in the memory 13. Here, the program includes the following instructions: extracting feature information about the facial region from the image including the facial region; transforming the extracted feature information to provide guidance for generating a portion within the facial region; and performing de-identification based on the provided guidance. The de-identification device 10 preferably pre-stores or sends at least one of the feature information extracted from the original image and the transformation rules for de-identification to the device 20 for re-identification of the de-identified facial image.

[0105] The re-identification device (or bioinformatics processing device) 20 includes a memory 23 and a processor 22. The memory 23 stores a program for receiving and de-identifying images, and the processor 22 executes the program stored in the memory 23. Here, the de-identified image is generated by: extracting feature information about a facial region from an original image; transforming the extracted feature information to provide guidance for the portion to be generated within the facial region; and performing de-identification based on the provided guidance. Furthermore, the program includes the following instructions: receiving the de-identified image; receiving feature information from the original image and at least one or a combination of 50 from a set of transformation rules for de-identification; and performing re-identification to recover a facial image from the input de-identified image using the feature information or transformation rules of the received original image.

[0106] exist Figure 9 In this process, the de-identification device 10 and the re-identification device 20 preferably send, receive, or share feature information of the original image and at least one or a combination of 50 from a set of transformation rules for de-identification. Therefore, the re-identification device 20 can recover the original facial image from the de-identified image.

[0107] The embodiments of this disclosure can be implemented by various means, such as hardware, firmware, software, or a combination thereof. When implemented in hardware, an embodiment of this disclosure can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc. When implemented in firmware or software, an embodiment of this disclosure can be implemented by modules, processes, functions, etc., that perform the above-described functions or operations. Software code can be stored in memory and can be driven by a processor. Memory can be located inside or outside the processor and can exchange data with the processor in various known ways.

[0108] Embodiments of this disclosure can be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data readable by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage devices, etc. Furthermore, computer-readable recording media can be distributed across computer systems connected via a network, and computer-readable code can be stored and executed in a distributed manner. The functional programs, code, and code segments used to implement embodiments of this disclosure can be readily explained by those skilled in the art to which this disclosure pertains.

[0109] According to various embodiments of this disclosure, de-identification can be performed by extracting region-specific features or transforming feature information obtained through grid-based masking processing. This can protect visual features within an image and retain partial information reflecting features from each region, thereby generating a natural face that is easily recognizable to humans. Furthermore, re-identification can be achieved by separately storing and using feature information or transformation rules to restore the visual features of the original image.

[0110] As described above, this disclosure has been studied in relation to various embodiments thereof. Those skilled in the art to which this disclosure pertains will understand that various embodiments may be implemented in modified forms within the scope of the essential features of this disclosure. Therefore, the disclosed embodiments are to be considered illustrative rather than restrictive. The scope of this disclosure is shown in the claims rather than in the foregoing description, and all differences within the scope should be construed as included in this disclosure.

Claims

1. A de-identification method, the de-identification method comprising: Receives images including facial regions; Extract feature information about the facial region from the input image; The extracted feature information is transformed to provide guidance for the portion to be generated within the facial region; and Based on the provided guidance, perform de-identification.

2. The de-identification method according to claim 1, further comprising performing at least one preprocessing step of shadow removal, noise removal, image modification, and color correction on the input image.

3. The de-identification method according to claim 1, wherein, The extraction of the feature information includes at least one of the following: Extracting region-specific features of different region sizes from the input image; and The input image is masked using grid cells of different sizes.

4. The de-identification method according to claim 3, wherein, The extraction of region-specific features includes: The entire image region is segmented into dedicated regions of different sizes. For each segmented region, features are extracted by transforming the high-dimensional image to a low-dimensional image, and the extracted features are then transformed into the latent space. During the transformation process of image data corresponding to a relatively large region, image information corresponding to a relatively small region is provided as input to each layer, and the input is adjusted to match the size of the larger region.

5. The de-identification method according to claim 3, wherein, Performing the masking process includes: At least two or more segmentation criteria are set, and the entire image region is divided into grids of predetermined size according to the set segmentation criteria, so as to extract the structural features of the corresponding image for each grid, and the grids segmented according to different segmentation criteria overlap in some regions; and The features extracted from each grid in each region are aggregated into structural features corresponding to the entire image region.

6. The de-identification method according to claim 5, wherein, Performing the masking process includes: Skin features are extracted from the relatively smaller grids within the segmented grid.

7. The de-identification method according to claim 1, wherein, The provided guidance includes: According to preset transformation rules, the feature information corresponding to the newly generated part in the facial area is removed or modified.

8. The de-identification method according to claim 7, further comprising pre-storing or sending at least one or a combination of the feature information and the transformation rule to a device configured to re-identify the de-identified facial image.

9. The de-identification method according to claim 1, wherein, Performing the de-identification includes: Generative artificial intelligence is used to generate de-identified facial images based on guidance provided corresponding to the parts to be newly generated.

10. The de-identification method according to claim 1, wherein, The extraction of the feature information includes: Using a diffusion model, parameters related to the positive transformation from a facial image to noise are learned, and Performing the de-identification includes: By performing an inverse transformation on the diffusion model based on the guidance of removing or modifying feature information, a de-identified facial image is generated from the noise.

11. The de-identification method according to claim 10, wherein, Performing the de-identification includes: During the inverse transformation of the diffusion model, the facial image is first generated using features extracted from images in relatively large grid regions and then using features extracted from images in progressively smaller grid regions.

12. A method for processing biometric information, the method comprising the following steps: Receive de-identified images; Receive feature information of the original image and at least one or a combination thereof from a set of transformation rules for de-identification; as well as Re-identification is performed using the feature information from the received original image or the received transformation rules to recover the facial image from the input deidentified image. The deidentified image is generated by: extracting feature information about the facial region from the original image; transforming the extracted feature information to provide guidance for the portion to be generated within the facial region; and performing deidentification based on the provided guidance.

13. The method according to claim 12, wherein, Performing the de-identification includes: The feature information of the original image is used to regenerate the feature information that has been removed or modified according to the set transformation rules, so as to restore the facial image.

14. The method according to claim 12, wherein, The de-identified image is generated in the following manner: For each specific region of different sizes from the input image, features of the corresponding image are extracted by transforming from a high-dimensional image to a low-dimensional image, and the extracted features are transformed into the latent space to extract region-specific features. or Masking is performed for each grid of different sizes from the input image to extract the structural features of the corresponding image, and the features extracted from each grid are aggregated for each region to extract structured feature information.

15. One or more non-transitory computer-readable media storing one or more instructions, wherein, The one or more instructions that can be executed by one or more processors are configured as follows: Receive an image including facial regions and perform de-identification; For each specific region of different sizes from the input image, features of the corresponding image are extracted by transforming from a high-dimensional image to a low-dimensional image, and the extracted features are transformed into the latent space to extract region-specific features, or masking is performed to extract structural features of the corresponding image from each grid of different sizes of the input image, and the features extracted from each grid are aggregated by region to extract structured feature information; The extracted feature information is transformed to provide guidance for the portion to be generated within the facial region; as well as Generative artificial intelligence is used to generate de-identified facial images based on guidance provided corresponding to the parts to be newly generated.

Images