Learning method of neural network for obfuscating face image and electronic device executing same

Abstract

A method of learning a neural network for obfuscating face images and an electronic device performing the same are disclosed. A method of learning a neural network for obfuscating a human face image according to an embodiment may include the following operations: acquiring an output human face image in which the input human face image is obfuscated based on the input human face image, and learning a neural network for obfuscating the input human face image based on the output human face image. Based on the input face image, extracting features of the input face image for restoring identity recognition information included in the input face image from the output face image; based on the output face image, extracting features of the output face image corresponding to the features of the input face image; and learning the neural network based on the difference between the features of the input face image and the features of the output face image.

Description

Technical Field

[0001] The following disclosure relates to a learning method for a neural network used to obfuscate facial images and an electronic device for performing the same. Background Technology

[0002] Image obfuscation is a technique that intentionally distorts or transforms an image to make the original image unrecognizable. There are two main metrics for evaluating obfuscated images. One is human indecipherability (HI), which represents the degree to which the obfuscated image is unrecognizable to humans. The other is machine decipherability (MD), which represents the effectiveness of the obfuscated image in being interpreted by machines (e.g., facial recognition algorithms).

[0003] Image obfuscation techniques can be used to hide specific parts of an image or protect the entire image, and can be applied in various fields such as information protection, privacy protection, and data security.

[0004] The above background technology was acquired or learned by the inventors during the development of this invention, and should not be construed as necessarily being known technology disclosed before the application for this invention. Summary of the Invention

[0005] Technical problems to be solved

[0006] One embodiment may provide a technique for learning neural networks to achieve a balance between the degree of obfuscation and the degree of restoration of the input face image.

[0007] However, the technical problems are not limited to those mentioned above; other technical problems may also exist.

[0008] Technical methods for solving problems

[0009] A method for learning a neural network for obfuscating facial images according to one embodiment may include the following operations: obtaining an output facial image that obfuscates the input facial image based on an input facial image; extracting features of the input facial image based on the input facial image for recovering identity information included in the input facial image from the output facial image; extracting features of the output facial image corresponding to the features of the input facial image based on the output facial image; and learning the neural network based on the differences between the features of the input facial image and the features of the output facial image.

[0010] The operation of obtaining the output face image may include the following operations: inputting the input face image into the neural network to generate the output face image.

[0011] The operation of generating the output face image may include the following operations: performing an averaging transformation on the input face image; rearranging the pixels of the averaged input face image by warping the averaged input face image; adding noise to the rearranged input face image; and generating the output face image by adjusting the color values ​​of the input face image with added noise.

[0012] The averaging transformation may include a mosaic transformation and a transformation that corrects the pixels on one axis of the image to the average value of the pixels.

[0013] The noise may include sinusoidal noise, checkerboard noise, and speckle noise.

[0014] Learning the operation of the neural network may include the following operation: updating the parameters of the neural network based on the differences between the features of the input face image and the features of the output face image using a backpropagation refinement scheme. The parameters of the neural network may be related to the obfuscation of the input face image.

[0015] The backpropagation refinement method can be a process of repeatedly executing forward and backward propagation processes to determine the parameters of the neural network, wherein the parameters of the neural network trade off between the degree of obfuscation of the output face image and the degree to which the identity information is recovered from the output face image. The forward propagation process may include operations such as acquiring the output face image, extracting features from the input face image, and extracting features from the output face image. The backpropagation process may include operations such as updating the parameters of the neural network.

[0016] Updating the parameters of the neural network may include the following operations: calculating the distance between the features of the input face image and the features of the output face image; and changing the parameters of the neural network to minimize the distance.

[0017] Updating the parameters of the neural network may include the following operations: calculating the cosine similarity between the features of the input face image and the features of the output face image; and changing the parameters of the neural network to maximize the cosine similarity.

[0018] The operation of updating the parameters of the neural network may include the following operation: changing the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold.

[0019] An electronic device for obfuscating a facial image according to one embodiment may include: a processor; and a memory storing instructions. When the instructions are executed by the processor, the instructions cause the electronic device to perform the following operations: based on an input facial image, obtain an output facial image (after obfuscation) from the input facial image using a neural network. The neural network may be learned using the method according to any one of claims 1 to 10.

[0020] An electronic device for learning a neural network for obfuscating face images according to one embodiment may include: a processor; and a memory storing instructions. When the instructions are executed by the processor, the instructions cause the electronic device to perform the following operations: based on an input face image, obtain an output face image that obfuscates the input face image; based on the input face image, extract features of the input face image for recovering identity information included in the input face image from the output face image; based on the output face image, extract features of the output face image corresponding to the features of the input face image; and learn the neural network based on the differences between the features of the input face image and the features of the output face image.

[0021] When the instructions are executed by the processor, the instructions can cause the electronic device to generate the output face image by inputting the input face image into the neural network.

[0022] When the instructions are executed by the processor, the instructions can cause the electronic device to perform the following operations: perform an average transformation on the input face image; rearrange the pixels of the input face image that has undergone the average transformation by deforming the input face image that has undergone the average transformation; add noise to the input face image whose pixels have been rearranged; and generate the output face image by adjusting the color values ​​of the input face image with the added noise.

[0023] The averaging transformation may include a mosaic transformation and a transformation that corrects the pixels on one axis of the image to the average value of the pixels.

[0024] When the instructions are executed by the processor, the instructions can cause the electronic device to perform the following operation: update the parameters of the neural network through backpropagation thinning based on the differences between the features of the input face image and the features of the output face image. The parameters of the neural network may be related to the obfuscation of the input face image.

[0025] The backpropagation refinement method can be a process of repeatedly executing forward and backward propagation processes to determine the parameters of the neural network, wherein the parameters of the neural network balance the degree of obfuscation of the output face image and the degree to which the identity information can be recovered from the output face image. The forward propagation process may include operations such as acquiring the output face image, extracting features from the input face image, and extracting features from the output face image. The backpropagation process may include operations such as updating the parameters of the neural network.

[0026] When the instructions are executed by the processor, the instructions can cause the electronic device to perform the following operations: calculate the distance between the features of the input face image and the features of the output face image; and change the parameters of the neural network to minimize the distance.

[0027] When the instructions are executed by the processor, the instructions can cause the electronic device to perform the following operations: calculate the cosine similarity between the features of the input face image and the features of the output face image; and change the parameters of the neural network to maximize the cosine similarity.

[0028] When the instruction is executed by the processor, the instruction can cause the electronic device to perform the following operation: change the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold. Attached Figure Description

[0029] Figure 1 An example of an electronic device for obfuscating facial images is shown according to one embodiment.

[0030] Figure 2 Show Figure 1 A schematic block diagram of the electronic device shown.

[0031] Figure 3 The accompanying drawings illustrate a method for learning a neural network by backpropagation refinement according to an embodiment.

[0032] Figure 4 An example flowchart illustrating a neural network learning method according to one embodiment is shown. Detailed Implementation

[0033] The specific structural or functional descriptions of the disclosed embodiments are for illustrative purposes only, and various modifications can be made to the embodiments. Therefore, the embodiments are not limited or restricted to the specific form of disclosure, and all variations, equivalents, or substitutions of the embodiments are included within the scope of the claims.

[0034] Furthermore, in the description of various components, terms such as "first" or "second" may be used, which are used only to distinguish one constituent element from another. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

[0035] When a constituent element is described as being “connected” to another constituent element, it should be understood that the constituent element can be directly connected to or attached to the other constituent element, or that the other constituent element is “connected” to the constituent elements.

[0036] Unless otherwise specified in the text, singular expressions include plural meanings. In this specification, terms such as "comprising" or "having" are used to indicate the presence of the features, numbers, steps, operations, constituent elements, accessories, or combinations thereof described in the specification, and do not exclude the presence of one or more other features, numbers, steps, operations, constituent elements, accessories, or combinations thereof, or additional functions.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, shall have the ordinary meaning as understood by one of ordinary skill in the art. Terms that are commonly used and are identical to their dictionary definitions shall be understood to have a meaning consistent with the general content of the relevant art, and shall not be overly idealized or interpreted as having a formal meaning unless expressly stated in this application.

[0038] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, in the description with reference to the drawings, the same reference numerals are used for the same constituent elements, and repeated descriptions thereof are omitted.

[0039] The module in this disclosure may be hardware capable of performing the functions and operations described herein, computer program code capable of performing specific functions and operations, or an electronic recording medium, such as a processor or microprocessor, loaded with computer program code capable of performing specific functions and operations.

[0040] In other words, a module can be a combination of hardware for performing the concepts and techniques described in this disclosure, and / or software for performing the hardware.

[0041] Figure 1 An example of an electronic device for obfuscating facial images is shown according to one embodiment.

[0042] Reference Figure 1 The electronic device 100 can learn a neural network (or a neural network model). Furthermore, the electronic device 100 can use the learned neural network to perform reasoning (e.g., scrambling facial images).

[0043] Neural networks (or artificial neural networks) can include statistical learning algorithms in the fields of machine learning and cognitive science that simulate biological neurons. A neural network can refer to a model in which artificial neurons (nodes) form a network through synaptic connections, and learn to change the strength of these synaptic connections, thereby gaining the ability to solve problems.

[0044] Neurons in a neural network can include weights and biases. A neural network can include more than one neuron or node and more than one layer. A neural network can learn to change the weights of its neurons, thus inferring a desired result from any input.

[0045] Neural networks can include deep neural networks (DNN). Neural networks can include: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Perceptrons, Multilayer Perceptrons (MLP), Feedforward Networks (FF), Radial Basis Function Networks (RBF), Deep Feedforward Networks (DFF), Long Short-Term Memory Networks (LSTM), Gated Recurrent Units (GRU), Autoencoders (AE), Variational Autoencoders (VAE), Denoising Autoencoders (DAE), Sparse Autoencoders (SAE), Markov Chains (MC), Hopfield Networks (HN), Boltzmann Machines (BM), Restricted Boltzmann Machines (RBM), Deep Belief Networks (DBN), Deep Convolutional Networks (DCN), Deconvolutional Networks (DN), Deep Convolutional Inverse Graph Networks (DCIGN), Generative Adversarial Networks (GAN), Liquid Machines (LSM), Extreme Learning Machines (ELM), Echo State Networks (ESN), Deep Residual Networks (DRN), Differentiable Neural Computers (DNC), Neural Turing Machines (NTM), Capsule Networks (CN), Kohonen Networks (KN), and Attention Networks (AN).

[0046] The electronic device 100 can be implemented on an embedded system with limited hardware resources by using a lightweight neural network model. The neural network learning device 10 can perform learning and inference on-device.

[0047] Electronic device 100 can be implemented as a printed circuit board (PCB) such as a motherboard, an integrated circuit (IC), or a system on chip (SoC). For example, electronic device 100 can be implemented as an application processor.

[0048] In addition, the electronic device 100 can also be implemented as a personal computer (PC), a data server, or a portable device.

[0049] Portable devices can be laptops, mobile phones, smartphones, tablets, mobile internet devices (MIDs), personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital cameras, digital camcorders, portable multimedia players (PMPs), personal navigation devices (PNDs) or portable navigation devices, handheld game consoles, e-readers, or smart devices. Smart devices can be smartwatches, smart bracelets, or smart rings.

[0050] Electronic device 100 can learn a neural network by processing the parameters (or weights) of the neural network model. Electronic device 100 can generate a lightweight neural network model by processing the parameters of a neural network model that has been learned with full precision.

[0051] The electronic device 100 can acquire new parameters by processing the parameters that change during the learning process of the neural network model, and retrain the neural network model based on the new parameters.

[0052] The electronic device 100 can obtain an obfuscated output facial image based on an input facial image.

[0053] The electronic device 100 can extract features from an input face image to reconstruct identity information included in the input face image from an output face image. The electronic device 100 can also extract features from an output face image that correspond to the features of the input face image.

[0054] The electronic device 100 can learn a neural network based on the differences between the features of the input face image and the features of the output face image. The neural network can be learned using a backpropagation refinement scheme. This will be discussed in more detail below. Figure 2 and Figure 3 To provide a more detailed description.

[0055] Electronic device 100 may include processor 110 and memory 120.

[0056] Processor 110 can process data stored in memory 120. Processor 110 can execute computer-readable code (e.g., software) stored in memory 120 and instructions triggered by processor 110.

[0057] The processor 110 can be a hardware-implemented data processing device with physically structured circuitry to perform desired operations. For example, the desired operations may include code or instructions included in a program.

[0058] For example, a hardware-implemented data processing device may include a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).

[0059] Memory 120 may store a neural network model or parameters of a neural network model. Memory 120 may store processor-executable instructions (or programs). For example, these instructions may include instructions for performing operations on the processor and / or on various components of the processor.

[0060] The memory 120 can be implemented as a volatile or non-volatile memory device.

[0061] Volatile memory devices can be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor random access memory (T-RAM), zero-capacitance random access memory (Z-RAM), or dual-transistor random access memory (TTRAM).

[0062] Non-volatile storage devices can be implemented as electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic random access memory (MRAM), spin-transfer torque magnetic random access memory (STT-MRAM), conductive bridged random access memory (CBRAM), ferroelectric random access memory (FeRAM), phase change random access memory (PRAM), resistive random access memory (RRAM), nanotube resistive random access memory (NRAM), polymer random access memory (PoRAM), nanofloating gate memory (NFGM), holographic memory, molecular electronic memory device, or insulator resistance change memory.

[0063] The processor 110 can cause the electronic device 100 to perform more than one operation by executing code and / or instructions stored in the memory 120. (Refer to the following...) Figures 2 to 4 Describe in detail the operations performed by the electronic device 100.

[0064] Figure 2 Show Figure 1 A schematic block diagram of the electronic device shown.

[0065] Reference Figure 2 The electronic device 100 may include an obfuscation module 210 and a feature extractor 230. The obfuscation module 210 and the feature extractor 230 may be implemented as different neural networks or as a single neural network. For ease of explanation, it is assumed below that the obfuscation module 210 and the feature extractor 230 are implemented as different neural networks.

[0066] The obfuscation module 210 can generate an output face image based on an input face image, wherein the input face image has been obfuscated. The obfuscation module 210 can perform various transformations (e.g., averaging transformation, warping, noise addition, and / or color value adjustment) sequentially or in parallel on the input face image to generate the output face image. These different transformations can be performed at different levels of the obfuscation module 210, and the degree of transformation can be determined according to the parameters of each level. The parameters of each level can be related to the degree of obfuscation of the input face image.

[0067] Typically, various transformations are applied to make the output face image more difficult for humans to recognize, and the degree of transformation can be set to a high value by default. If the degree of transformation is set high, the level of obfuscation may increase; however, it may become difficult to recover the input face image from the output face image. Therefore, it may be necessary to set the parameters of the layers of the obfuscation module 210 in a way that balances the level of obfuscation of the output face image and the level of recovery of the input face image.

[0068] An input face image can be reconstructed from an output face image by utilizing the differences (or similarities) between the features of the input face image and the features of the output face image. Reconstructing the input face image from the output face image can include restoring the identity information included in the input face image. As the differences between the features of the input face image and the features of the output face image decrease, it becomes easier to restore the identity information included in the input face image from the output face image. For example, when the input face image is transformed by the obfuscation module 210, the smaller the difference between the features of the input face image and the features of the transformed image (e.g., the output face image), the higher the reconstructibility (e.g., the degree to which the identity information included in the input face image is restored from the output face image). To this end, the features of the input face image and the features of the output face image can be extracted separately by the feature extractor 230, and the parameters of the layers of the obfuscation module 210 can be learned to minimize the difference between these two features.

[0069] Feature extractor 230 can extract features from a face image (e.g., an input face image and / or an output face image). For example, feature extractor 230 can extract features from the input face image to recover the identification information included in the input face image from the output face image. Feature extractor 230 can also extract features of the output face image corresponding to the features of the input face image based on the output face image.

[0070] The facial image features extracted by feature extractor 230 can be determined based on information to be restored from the input facial image (e.g., the identity information of the input facial image). For example, if the gender information of the input facial image is to be restored from the output facial image, feature extractor 230 can extract features related to the gender of the input facial image.

[0071] By learning the parameters of the layers of the confusion module 210 based on the differences between the features of the input face image and the features of the output face image, the reproducibility (or degree of reproducibility) of the input face image from the output face image generated by the confusion module 210 can be improved.

[0072] Electronic device 100 can update the parameters of the layers of obfuscation module 210 (e.g., layers performing various transformations) to achieve a trade-off between obfuscation and restoration. This is likely because stronger obfuscation results in weaker restoration, and vice versa; therefore, a trade-off between obfuscation and restoration is particularly important. The parameters of the layers of obfuscation module 210 can be updated by learning using a backpropagation refinement method, which will refer to... Figure 3 Provide a detailed description.

[0073] Figure 3 The accompanying drawings illustrate a method for learning a neural network by backpropagation refinement according to an embodiment.

[0074] Reference Figure 3 The electronic device 100 may include a parameter initialization module 310, an obfuscation module 210, and a feature extractor 230.

[0075] The obfuscation module 210 can process the input face image Perform various transformations (e.g., averaging, warping, noise addition, and / or color value adjustment). These transformations can be performed by multiple layers included in the obfuscation module 210, and the transformations performed by each layer will be described below.

[0076] The obfuscation module 210 may include an averaging layer 320, a warping layer 330, a noising layer 340, and a scaling layer 350.

[0077] An average layer of 320 can process the input face image Perform an averaging transformation. An averaging transformation can be a transformation that removes high-frequency information from the input face image while preserving low-frequency information, thereby removing image details. An averaging transformation can include a mosaic transformation. A transformation that adjusts the pixels on one axis of the image to the average of the aforementioned pixels (e.g., horizontal mean transformation). and / or vertical mean transformation ).

[0078] Mosaic Transformation Input face image The image is divided into multiple blocks, and the pixel values ​​of each block are adjusted to the average of the pixel values ​​within that block. For example, averaging layer 320 can divide the image into M×N blocks and calculate the average pixel value of each block. Averaging layer 320 can adjust the pixel values ​​of each block to the average of the pixel values ​​within that block. As a result, all pixel values ​​within each M×N block can become identical, thereby removing high-frequency information from the input face image. Mosaic Transform The degree can be based on the parameters of an average layer of 320. Determined. For example, parameters. The larger the value, the greater the degree of mosaic transformation, which may result in a poorly rendered output face image. The level of confusion is even higher.

[0079] Horizontal average transformation Input face image The pixel values ​​of the blocks arranged along the horizontal axis are adjusted to the average of the pixel values ​​within each block. This can be done based on the parameters of the averaging layer 320. Determine the average transformation For example, parameters The larger the horizontal average transformation The higher the degree, the more likely it is to increase the output face image. The degree of confusion.

[0080] Vertical average transformation Input face image The pixel values ​​of the blocks arranged along the vertical axis are adjusted to the average of the pixel values ​​within each block. This can be done based on the parameters of the averaging layer 320. Determine the vertical average transformation For example, parameters The larger the vertical average transformation The higher the degree, the more likely it is to increase the output face image. The degree of confusion.

[0081] An average layer of 320 can be combined with an already performed mosaic transformation. Horizontal averaging transformation and vertical average transformation The input face image. For example, an average layer of 320 can be processed using the SoftMax operation to perform a mosaic transformation. Horizontal averaging transformation and vertical average transformation The input face image is normalized by blocks. Then, the averaging layer 320 can combine the normalized blocks to generate a single overlapping block. The image after averaging transformation can include multiple overlapping blocks. Specifically, the averaging layer 320 can combine the mosaic transform already performed using the following mathematical expression 1. Horizontal averaging transformation and vertical average transformation The input is a human face image.

[0082] [Mathematical Expression 1]

[0083]

[0084] In mathematical formula 1, This indicates that a mosaic transformation has been performed. Image blocks, Indicates that a horizontal averaging transformation has been performed. Image blocks, This indicates that a vertical averaging transformation has been performed. Image blocks, to This represents the random parameter used to perform the SoftMax operation on the corresponding index block. represents overlapping blocks, and c represents the RGB channels of the input face image.

[0085] Deformation layer 330 can perform deformation transformation (hereinafter referred to as deformation transformation) on the input face image that has undergone average transformation. This rearranges the pixels of the input face image, which has undergone an average transformation. That is, a deformation transformation. This can include transformations of image geometric features. For example, deformation layer 330 can deform a face image based on grid points (e.g., intersections between blocks) within M×N blocks of the face image (e.g., an input face image that has been averaged and transformed by averaging layer 320). Deformation layer 330 can shift each grid point by a parameter-based transformation layer 330. The value. Deformation layer 330 can move each grid point ( Here, ∆ represents the size of the block (e.g., the vertical size of the block when a grid point is moved vertically) (e.g., the horizontal size of the block when a grid point is moved horizontally). The deformation layer 330 can move grid points in a manner that avoids grid points overlapping each other. Thus, the deformation transformation... This may cause distortion of facial images. (The last part, "with parameters," appears to be incomplete and unrelated to the preceding text. With the increase in the number of grid points, more grid points move, making the deformation transformation... The increased degree of [something] may increase the output face image. The degree of confusion.

[0086] Noise layer 340 can add noise to an input face image whose pixels have been rearranged (e.g., an input face image transformed by averaging layer 320 and deformation layer 330). By adding noise, noise layer 340 can introduce high-frequency components to increase the complexity of the image, thereby improving the reproducibility of the original image (e.g., the input face image) from the transformed image.

[0087] Noise can include sinusoidal noise, checkerboard noise, and / or speckle noise. For example, noise layer 340 can perform a sinusoidal noise-adding transformation on a face image. Noise layer 340 can perform a checkerboard-based noise-adding transformation on the face image. Noise layer 340 can perform a speckle-based noise-adding transformation on the face image. .

[0088] Noise Addition Transformation Based on Sine Curve It can be based on the parameters of noise layer 340 Execution. For example, noise layer 340 can be applied to each block of a face image (e.g., an input face image transformed by averaging layer 320 and deformation layer 330) along parameters. The axis generates sine-based noise. Noise layer 340 can enhance high-frequency components by adding sine-based noise (e.g., periodic patterns) to the face image.

[0089] Noise Addition Transformation Based on Checkerboard Pattern It can be based on the parameters of noise layer 340 Execution. For example, noise layer 340 can add NxN (e.g., 4x4) checkerboard patterns to each block of the face image. This can be determined based on parameters. Adjust NxN checkerboard patterns. Adding NxN checkerboard patterns may result in high-frequency components being added to the face image.

[0090] Spot-based noise addition transformation It can be based on the parameters of noise layer 340 Execution. For example, noise layer 340 can assign parameters. The center of each block in the face image is assigned. Noise layer 340 can be obtained from the assigned parameters. Bilinear interpolation is performed at the center of each block to determine the pixel values ​​of the remaining blocks.

[0091] Noise layer 340 can add transformations to the already performed sine-based noise. Noise Addition Transformation Based on Checkerboard Pattern and speckle-based noise addition transformation The face images are combined. For example, noise layer 340 can use the SoftMax operation to add a transformation to the noise in the input face image that has already undergone a sinusoidal curve transformation. Noise Addition Transformation Based on Checkerboard Pattern and speckle-based noise addition transformation The blocks are normalized. Noise layer 340 can then normalize the blocks and perform a deformation transformation on them using deformation layer 330. The image patches are combined to generate a single overlapping block. The image to which the noise-adding transformation is performed may include multiple overlapping blocks. Specifically, the noise layer 340 can perform a sine-based noise-adding transformation on the image using the following mathematical formula 2. Noise Addition Transformation Based on Checkerboard Pattern and speckle-based noise addition transformation The facial images are combined.

[0092] [Mathematical Expression 2]

[0093]

[0094] In mathematical formula 2, Indicates that the transformation has been completed. Image blocks, This indicates that the noise addition transformation based on the sine function has been completed. Image blocks, This indicates that the noise addition transformation based on the checkerboard function has been completed. Image blocks, This indicates that the noise addition transformation based on the blob function has been performed. Image blocks, to This represents the random parameter used in the SoftMax operation for the block at the corresponding index. represents overlapping blocks, and c represents the RGB channels of the input face image.

[0095] The scaling layer 350 can be adjusted (hereinafter referred to as scaling transformation). The output face image is generated by adding color values ​​to a noisy face image (e.g., an input face image that has undergone transformation by averaging layer 320, warping layer 330, and noise layer 340). Scaling transformation. It can be a transformation that adjusts the color values ​​(e.g., color intensity) of a face image.

[0096] Scaling transformation It can be based on the parameters of scaling layer 350 To execute. For example, scaling transformation. Parameters can be used This is used to adjust the color values ​​of each region in the face image. When the parameter... When the value is greater than 1, the brightness of the block will increase; when the parameter... When the value is less than 1, the brightness of the block will decrease. This is achieved by adjusting the color intensity and scaling the image. It can increase the level of obfuscation while increasing the reproducibility (e.g., the degree of reproducibility of the original face image (e.g., the input face image) from the transformed face image (e.g., the output face image).

[0097] Feature extractor 230 may include multiple feature extractors 360 and 370. Feature extractor 360 can extract features from the output face image. Feature extractor 370 can extract features from input face images. The features of feature extractors 360 and 370 are essentially the same as those of feature extractor 230, and therefore will not be described further.

[0098] The above describes in detail how the input face image is processed through multiple layers included in the obfuscation module 210. Perform transformations to obtain (or generate) an output face image. The method involves inputting a face image. The transformation can be based on parameters of multiple layers included in the obfuscation module 210 (e.g., parameters). to This can be done based on parameters. to The update (or learning) of the input face image Perform a transformation to determine the output face image The degree of confusion and recoverability. The following section details a method for learning neural networks using backpropagation refinement to determine the level of confusion and recoverability in the output face image. A parameter that balances the degree of confusion and the degree of restoration (e.g., the degree to which the identity information of the input face image is restored from the output face image). to .

[0099] First, the difference between the backpropagation refinement method and the traditional backpropagation method will be described. The traditional backpropagation method calculates the loss function by comparing the output of the forward propagation process with the ground truth, and then updates the parameters of the neural network through the backpropagation process. However, the backpropagation refinement method updates the parameters of the neural network by repeatedly performing the forward and backpropagation processes, instead of calculating the loss function by comparing with the ground truth. The forward propagation process may include operations performed by the confusion module 210 (e.g., obtaining the output face image based on the input face image) and operations performed by the feature extractor 230 (e.g., extracting features from the input face image and extracting features from the output face image). The backpropagation process may include updating the parameters of the neural network.

[0100] The method for updating neural network parameters by backpropagation refinement according to the present invention will now be described in detail.

[0101] The parameters of the obfuscation module 210 (e.g., parameters) to It can control the transformation of the input face image, and can determine the quality of the output face image based on the initialization method of these parameters.

[0102] Parameters can be categorized into four types based on their characteristics. The parameter initialization module 310 can use different initialization methods to initialize parameters of different categories.

[0103] Based on their characteristics, parameters can include fixed parameters, uniform parameters, color parameters, and composing parameters. The characteristics and initialization methods for each type of parameter will be described below.

[0104] Fixed parameters refer to parameters that are fixed to specific values. Fixed parameters can be used to remove high-frequency information (or details) from facial images, thereby increasing the obfuscation level (or human indecipherability) of the face image. Due to the parameters... to and This is used to remove high-frequency information from face images, so it can be a fixed parameter. to and It can be fixed to a specific value (e.g., 1). For example, when based on the parameter... (For example, 1) Perform a mosaic transformation At this time, all pixels in each block can be adjusted to the same average value, thereby removing high-frequency information from the input face image.

[0105] Since the fixed parameters always have fixed values, the parameter initialization module 310 does not need to be initialized separately.

[0106] A uniform parameter can refer to a parameter that is uniformly distributed within a specific range. , and It can be a uniform parameter.

[0107] The parameter initialization module 310 can set parameters. , and The value, so that it is within a specific range (e.g., to ) has a uniformly distributed value (e.g., For example, in deformation transformation In the middle, the parameter initialization module 310 can be based on the parameters The movement range of the grid points is set to [-0.3, 0.3]. The parameter initialization module 310 can set the parameters. The initial value is (e.g., 10 / 6). Since the grid points move based on values ​​uniformly distributed in the range [-0.3, 0.3] (e.g., 10 / 6), sufficient deformation can be induced in the face image without causing excessive grid distortion. This is due to the noise-added transformation based on the sine curve. and speckle-based noise addition transformation Initialization method and deformation transformation Similarly, so I won't go into details.

[0108] Color parameters are parameters that control color transformation and can be used to adjust the colors of an input face image. Because of these parameters... Used to adjust the color intensity of an image, therefore it can be a color parameter. When the parameter... When the value is greater than 1, the brightness of the input face image will increase; when the parameter... When the value is less than 1, the brightness of the input face image will decrease.

[0109] The parameter initialization module 310 can set parameters by determining whether brightness increases or decreases with equal probability (e.g., 50%). The value. The parameter initialization module 310 can be within a specific range (e.g., to Initialization parameters In this way, color transformation can obfuscate images while preserving high-frequency information.

[0110] Combination parameters can refer to parameters used to combine the results of multiple transformations. Combination parameters can be used to combine transformed image patches. Combination parameters can include random parameters used in the SoftMax operation (e.g., in Equation 1). to and in mathematical formula 2 to For example, the combination parameters can be normalized using the SoftMax operation, thereby appropriately adjusting the proportion of the combination blocks.

[0111] The parameter initialization module 310 can uniformly initialize the combined parameters within a specific range (e.g., 0 to 1).

[0112] The parameters of the neural network (e.g., the obfuscation module 210) initialized by parameter initialization module 310 can be learned through backpropagation refinement as described below, thereby determining a balance between the degree of obfuscation of the output face image and the degree to which the identification information included in the input face image is recovered from the output face image. The method for updating (or optimizing) the initialization parameters will be described in detail below.

[0113] The obfuscation module 210 can be configured based on parameters (e.g., parameters) of multiple layers (e.g., averaging layer 320, deformation layer 330, noise layer 340, and scaling layer 350). to And combined parameters (e.g., in mathematical formula 1) to and in mathematical formula 2 to The initial values ​​of the input face image are used to transform the image. This allows us to obtain (or generate) the output face image. Even if the output face image is generated based on the initial parameter values. It has a high degree of obfuscation and its reproducibility (e.g., from the output face image) Restore the input face image The level of identity verification information may also be low. For example, the output face image Features (e.g., extracted by feature extractor 360) can be compared with the input face image. The features (e.g., those extracted by feature extractor 370) show significant differences.

[0114] Four loss functions can be used to learn the parameters of the confusion module 210.

[0115] Fixed parameters (e.g., parameters) to and The value is fixed, so it does not need to be updated.

[0116] The electronic device 100 can modify the parameters of the neural network so that the parameters do not exceed a predetermined threshold (e.g., outer margin). Specifically, the uniformity parameter and color parameter can be learned by the electronic device 100, as described below.

[0117] The first loss function can be used to learn uniform parameters (e.g., parameters). and and The first loss function updates the uniform parameters by increasing the degree of confusion. The first loss function can be represented by the following mathematical formula 3.

[0118] [Mathematical Expression 3]

[0119]

[0120] In mathematical formula 3, Represents the uniformity parameter. Represented as parameters Set the outer margin (e.g., parameter) 0.05, parameter The parameter is 0. (0.1).

[0121] The first loss function can be set as follows: the closer the uniform parameter is to the outer margin, the greater the degree of confusion.

[0122] Color parameters (e.g., parameters) can be learned using a second loss function. The second loss function can update the color parameters by increasing the degree of confusion. The second loss function can be represented by the following mathematical formula 4.

[0123] [Mathematical Expression 4]

[0124]

[0125] In mathematical expression 4, [ ] represents an indicator function. Indicates color parameters, Represented as parameters Set the outer margin (e.g., for parameters) (Take 1.05).

[0126] The second loss function can determine the direction of color parameter optimization, whether to increase or decrease brightness.

[0127] To improve the reproducibility of the output face image (e.g., the degree to which the identification information included in the input face image is recovered from the output face image), the parameters of the neural network (e.g., parameters) can be learned through a third loss function and a fourth loss function. up to θ_8 Combination parameters (e.g., in mathematical formula 1) to and in mathematical formula 2 to ).

[0128] Electronic device 100 can calculate the distance (e.g., Euclidean distance) between features of the input face image and the output face image. Electronic device 100 can calculate a third loss function based on the Euclidean distance. The third loss function can be configured to minimize the output face image. Features and input face image The Euclidean distance between the features is calculated. Electronic device 100 can update the parameters of the neural network to minimize this distance. Specifically, the third loss function can be represented by the following mathematical formula 5.

[0129] [Mathematical Expression 5]

[0130]

[0131] In mathematical formula 5, This indicates that the output is a face image. Features Indicates the input face image Its characteristics.

[0132] The electronic device 100 can calculate the cosine similarity between features of the input face image and the output face image. Based on the cosine similarity, the electronic device 100 can calculate a fourth loss function. The fourth loss function can be configured to maximize the output face image. Features and input face image The cosine similarity between features is calculated. Electronic device 100 can update the parameters of the neural network to maximize the cosine similarity. Specifically, the fourth loss function can be represented by the following mathematical formula 6.

[0133] [Mathematical Expression 6]

[0134]

[0135] In mathematical formula 6, This indicates that the output is a face image. Features Indicates the input face image Its characteristics.

[0136] Therefore, the first to fourth loss functions can be learned in an end-to-end manner using the loss functions in the following mathematical formula 7.

[0137] [Mathematical Expression 7]

[0138]

[0139] The loss function in Formula 7 is a weighted sum of the first to fourth loss functions, which can be set to find the optimal parameters for balancing confusion and reproducibility.

[0140] These parameters can be learned through backpropagation, thereby optimizing the loss function obtained through forward propagation. Specifically, electronic device 100 can calculate the derivative of the loss function and the gradient of each parameter. The derivative of the loss function can be expressed by the following mathematical expressions 8 to 10.

[0141] [Mathematical Expression 8]

[0142]

[0143] [Mathematical Expression 9]

[0144]

[0145] [Mathematical Expression 10]

[0146]

[0147] The parameters in mathematical expressions 8 to 10 are the same as those in mathematical expressions 1 to 7.

[0148] Electronic device 100 can update parameters based on gradients. This process (e.g., calculating the loss function via forward propagation, calculating the derivative of the loss function via backpropagation, and updating the parameters accordingly) can be repeated until convergence, thereby determining the parameters of a neural network (e.g., obfuscation module 210) that balances the degree of obfuscation and restoration of the output face image (e.g., the degree to which identity information is restored from the output face image).

[0149] Learning is performed through backpropagation refinement, so that even after parameter initialization, the output face image can still be recognized by machine learning algorithms, while making it difficult for people to recognize the output face image.

[0150] Figure 4 An example flowchart illustrating a neural network learning method according to one embodiment is shown.

[0151] Reference Figure 4 Operations 410 to 470 can be executed sequentially, but are not limited to this. For example, two or more operations can be executed in parallel. Operations 410 to 470 can be related to a reference... Figures 1 to 3 The described electronic device (e.g., Figure 1 The operation of the electronic device 100 is substantially the same. Therefore, its detailed description will be omitted.

[0152] In operation 410, electronic device 100 can acquire an obfuscated output facial image based on the input facial image.

[0153] In operation 430, electronic device 100 can extract features of the input face image based on the input face image for reconstructing the identity information included in the input face image from the output face image.

[0154] In operation 450, the electronic device 100 can extract features of the output face image that correspond to the features of the input face image based on the output face image.

[0155] In operation 470, electronic device 100 can learn a neural network based on the difference between the features of the input face image and the features of the output face image.

[0156] The embodiments described above can be implemented using hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in the embodiments can be implemented using, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field-programmable array (FPA), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, and can be embodied using more than one general-purpose computer or special-purpose computer. The processing device can execute an operating system (OS) and more than one application software running within said operating system. Furthermore, the processing device responds to the execution of the software, thereby accessing, storing, manipulating, processing, and generating data. For ease of understanding, the description is presented as having only one processing device, but those skilled in the art will understand that a processing device can include multiple processing elements and / or multiple types of processing elements. For example, a processing device can include multiple processors or one processor and one controller. Furthermore, it can also include other processing configurations similar to parallel processors.

[0157] Software can include computer programs, code, instructions, or a combination of more than one of these, enabling a processing device to operate in a desired manner, or to command the processing device individually or collectively. To interpret or provide commands or data to the processing device, the software and / or data can be permanently or temporarily embodied in any type of device, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave. Software is distributed across a network-connected computer system and can be stored or executed in a distributed manner. Software and data can be stored on more than one computer read / write storage medium.

[0158] The method according to the embodiments is embodied in the form of program instructions executable by various computer means and recorded in a computer read / write medium. The computer read / write medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be instructions specifically designed and configured to implement the embodiments, or instructions that can be used by a person skilled in the art of computer software based on commonly known methods. The computer read / write recording medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices specifically configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), and flash memory. Examples of program instructions include not only machine language code generated by a compiler, but also high-level language code executable by a computer using an interpreter or similar means.

[0159] The hardware device described above can be configured to act as one or more software modules to perform the operations of the embodiments described above, and vice versa.

[0160] In summary, the embodiments have been described with reference to the limited accompanying drawings. Those skilled in the art can make various modifications and variations based on the description. For example, appropriate results can be obtained by performing the described techniques in a different order than the described methods, and / or by combining or integrating the described constituent elements in a different manner than the described methods, or by replacing or substituting them with other constituent elements or equivalents.

[0161] Therefore, other embodiments, other implementations, and equivalents of the claims are all within the scope of the appended claims.

Claims

1. A method for learning a neural network for obfuscating a face image, the method comprising: obtaining, based on an input face image, an output face image obfuscated from the input face image; extracting, based on the input face image, a feature of the input face image for recovering identity information included in the input face image from the output face image; extracting, based on the output face image, a feature of the output face image corresponding to the feature of the input face image; and learning the neural network based on a difference between the feature of the input face image and the feature of the output face image. 2.The method of claim 1, wherein the obtaining the output face image comprises: inputting the input face image into the neural network to generate the output face image. 3.The method of claim 2, wherein the generating the output face image comprises: performing an average transformation on the input face image; rearranging pixels of the input face image having undergone the average transformation by deforming the input face image having undergone the average transformation; adding noise to the input face image having undergone the rearrangement of the pixels; and generating the output face image by adjusting color values of the input face image having undergone the addition of the noise. 4.The method of claim 3, wherein the average transformation comprises a mosaic transformation and a transformation that corrects pixels on one axis of an image to an average value of the pixels. 5.The method of claim 3, wherein the noise comprises a noise based on a sinusoidal curve, a noise based on a checkerboard, and a noise based on a spot. 6.The method of claim 2, wherein the learning the neural network comprises: updating parameters of the neural network based on the difference between the feature of the input face image and the feature of the output face image by a backpropagation refinement method, wherein the parameters of the neural network are related to obfuscation of the input face image. 7.The method of claim 6, wherein the backpropagation refinement method is a method of repeatedly performing a forward propagation process and a backward propagation process to determine the parameters of the neural network, the parameters of the neural network being balanced between a degree of obfuscation of the output face image and a degree of recovery of the identity information from the output face image, wherein the forward propagation process comprises the operations of obtaining the output face image, extracting the feature of the input face image, and extracting the feature of the output face image, the backward propagation process comprises the operation of updating the parameters of the neural network. 8.The method of claim 7, wherein the updating the parameters of the neural network comprises: calculating a distance between the feature of the input face image and the feature of the output face image; and changing the parameters of the neural network to minimize the distance. 9.The method of claim 7, wherein ​ ​ ​ ​ ​ ​ ​ ​ ​ updating the parameters of the neural network, including the following operations: calculating a cosine similarity between the features of the input face image and the features of the output face image; and changing the parameters of the neural network to maximize the cosine similarity.

10. The method of claim 7, wherein updating the parameters of the neural network, including the following operations: changing the parameters of the neural network to make the parameters of the neural network not exceed a preset threshold.

11. An electronic device for obfuscating a face image, comprising: a processor; and a memory storing instructions, wherein when the instructions are executed by the processor, the instructions cause the electronic device to perform the following operations: obtaining, by a neural network, an output face image in which the input face image has been obfuscated, based on an input face image, wherein the neural network is learned by the method of claim 1.

12. An electronic device for learning a neural network for obfuscating a face image, comprising: a processor; and a memory storing instructions, wherein when the instructions are executed by the processor, the instructions cause the electronic device to perform the following operations: obtaining an output face image obfuscating the input face image, based on an input face image, extracting, based on the input face image, features of the input face image for recovering identity information included in the input face image from the output face image, extracting, based on the output face image, features of the output face image corresponding to the features of the input face image, learning the neural network based on a difference between the features of the input face image and the features of the output face image.

13. The electronic device of claim 12, wherein when the instructions are executed by the processor, the instructions cause the electronic device to generate the output face image by inputting the input face image to the neural network.

14. The electronic device of claim 13, wherein when the instructions are executed by the processor, the instructions cause the electronic device to perform the following operations: performing an average transformation on the input face image, rearranging pixels of the input face image having performed the average transformation by deforming the input face image having performed the average transformation, adding noise to the input face image whose pixels have been rearranged, generating the output face image by adjusting color values of the input face image having added the noise.

15. The electronic device of claim 14, wherein the average transformation includes a mosaic transformation and a transformation that corrects pixels on one axis of an image to an average value of the pixels.

16. The electronic device of claim 13, wherein when the instructions are executed by the processor, the instructions cause the electronic device to perform the following operations: updating the parameters of the neural network by a backpropagation refinement method based on a difference between the features of the input face image and the features of the output face image, ​ ​ The parameters of the neural network are related to the confusion of the input face image. 17.The electronic device of claim 16, wherein, The back propagation refinement manner is a method of repeatedly performing a forward propagation process and a back propagation process to determine the parameters of the neural network, the parameters of the neural network being a trade-off between a degree of confusion of the output face image and a degree of recovering the identity information from the output face image, The forward propagation process includes operations of obtaining the output face image, extracting a feature of the input face image, and extracting a feature of the output face image. The back propagation process includes an operation of updating the parameters of the neural network. 18.The electronic device of claim 17, wherein, When the instructions are executed by the processor, the instructions cause the electronic device to perform operations of: calculating a distance between the feature of the input face image and the feature of the output face image, changing the parameters of the neural network to minimize the distance. 19.The electronic device of claim 17, wherein, When the instructions are executed by the processor, the instructions cause the electronic device to perform operations of: calculating a cosine similarity between the feature of the input face image and the feature of the output face image, changing the parameters of the neural network to maximize the cosine similarity. 20.The electronic device of claim 17, wherein, When the instructions are executed by the processor, the instructions cause the electronic device to perform operations of: changing the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold.

Images