A method for training a neural network to randomize facial images and an electronic device for performing this training.

Abstract

This invention provides a method for training a neural network to randomize facial images and an electronic device for performing this training. [Solution] A method for training a neural network to randomize face images includes: generating an output face image based on an input face image, extracting features of the input face image from the output face image to reconstruct identity information contained in the input face image, based on the input face image, based on the output face image, based on the output face image, based on the output face image, based on the output face image, based on the output face image, based on the difference between the features of the input face image and the features of output face image, based on the difference between the features of the input face image and the output face image, based on the difference between the features of the input face image and the output face image, based on the difference between the input face image and the output face image, based on the input face image, based on the input face image, based on the input face image, based on the input face image, based on the input face image, based on the output face image, based on the input face image, based on the input face image, based on the output face image, based on the input face image, based on the input face image, based on the output face image, based on the input face image, based

Description

[Technical Field]

[0001] The following disclosure relates to a method for training a neural network for randomizing facial images and an electronic apparatus for performing the same. [Background technology]

[0002] Image obfuscation is a technique that intentionally distorts or deforms an image to make it unrecognizable from the original. Image obfuscation can be evaluated using two main metrics. One is human indecipherability (HI), which indicates how unrecognizable the image is to humans. The other is machine decipherability (MD), which indicates how efficiently the image can be deciphered by a machine (e.g., a facial recognition algorithm).

[0003] Image roguelike technology is used to hide specific parts of an image or to protect the entire image, and can be utilized in various application areas for information protection, privacy protection, or data security.

[0004] The background technology described above is something the inventors have acquired or learned in the process of deriving the disclosures in this specification, and is not necessarily publicly known technology that was made public before this application was filed. [Overview of the project] [Problems that the invention aims to solve]

[0005] One embodiment aims to provide a technique for training a neural network to strike a good balance between the degree of randomization of input face images and the degree of restoration to the input face images.

[0006] However, the technical challenges are not limited to those mentioned above; other technical challenges exist. [Means for solving the problem]

[0007] A method for training a neural network that randomizes facial images according to one embodiment may include: obtaining an output facial image in which the input facial image has been randomized based on an input facial image; extracting features of the input facial image from the output facial image to reconstruct identity information contained in the input facial image based on the input facial image; extracting features of the output facial image corresponding to the features of the input facial image based on the output facial image; and training the neural network based on the difference between the features of the input facial image and the features of the output facial image.

[0008] The operation to acquire the output face image may include the operation of inputting the input face image into the neural network and generating the output face image.

[0009] The operation to generate the output face image may include: performing an averaging transformation on the input face image; warping the input face image after the averaging transformation to rearrange the pixels of the input face image after the averaging transformation; adding noise to the input face image whose pixels have been rearranged; and adjusting the color values ​​of the input face image to which the noise has been added to generate the output face image.

[0010] The averaging transformation may include a mosaic transformation and a transformation that corrects pixels included in one axis of the image to the average of those pixels.

[0011] The noise can include noise on a sinusoid basis, noise on a checkerboard basis, and noise on a speckle basis.

[0012] The operation of training the neural network can include an operation of updating parameters of the neural network via a backpropagation refinement scheme based on a difference between features of the input face image and features of the output face image. The parameters of the neural network relate to randomization of the input face image.

[0013] The backpropagation refinement scheme is a scheme that repeatedly performs a forward propagation process and a backpropagation process in order to determine parameters of the neural network in which a degree of randomization of the output face image and a degree of restoration of the identity information from the output face image are traded off. The forward propagation process can include an operation of obtaining the output face image, an operation of extracting characteristics of the input face image, and an operation of extracting features of the output face image. The backpropagation process can include an operation of updating parameters of the neural network.

[0014] The operation of updating parameters of the neural network can include an operation of calculating a distance between features of the input face image and features of the output face image, and an operation of changing parameters of the neural network so that the distance becomes minimal.

[0015] [[ID=*15]] The operation of updating parameters of the neural network can include an operation of calculating a cosine similarity between features of the input face image and features of the output face image, and an operation of changing parameters of the neural network so that the cosine similarity becomes maximal.

[0016] Note: There seems to be a duplicate tag <ID=15> in the original text which is preserved as is in the translation.The operation of updating the parameters of the neural network may include an operation of changing the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold value.

[0017] An electronic device for obfuscating a face image according to an embodiment may include a processor and a memory for storing instructions. When the instructions are executed by the processor, the electronic device can obtain an output face image in which an input face image is obfuscated via a neural network based on the input face image (input facial image). The neural network is learned by the method according to any one of Items 1 to 10.

[0018] An electronic device for training a neural network for obfuscating a face image according to an embodiment may include a processor and a memory for storing instructions. When the instructions are executed by the processor, the electronic device can obtain an output face image in which an input face image is obfuscated based on the input face image (input facial image), extract features of the input face image for restoring identity information included in the input face image from the output face image based on the input face image, extract features of the output face image corresponding to the features of the input face image based on the output face image, and train the neural network based on a difference between the features of the input face image and the features of the output face image.

[0019] When the instructions are executed by the processor, the electronic device can input the input face image into the neural network and generate the output face image.

[0020] When the aforementioned instruction is performed by the processor, the electronic device performs an averaging transformation on the input face image, warps the input face image after the averaging transformation, rearranges the pixels of the input face image after the averaging transformation, adds noise to the input face image with rearranged pixels, adjusts the color values ​​of the input face image with the added noise, and generates the output face image.

[0021] The averaging transformation may include a mosaic transformation and a transformation that corrects pixels included in one axis of the image to the average of those pixels.

[0022] When the aforementioned instruction is executed by the processor, the electronic device can update the parameters of the neural network via a backpropagation refinement scheme based on the difference between the features of the input face image and the features of the output face image. The parameters of the neural network relate to the randomization of the input face image.

[0023] The aforementioned backpropagation purification method is a method that repeatedly performs a forward propagation process and a backpropagation process in order to determine the parameters of the neural network in which the degree of randomization of the output face image and the degree of identity information reconstruction from the output face image are traded off. The forward propagation process is the process of acquiring the output face image, extracting the characteristics of the input face image, and extracting the features of the output face image. The backpropagation process is the process of updating the parameters of the neural network.

[0024] When the aforementioned instruction is executed by the processor, the electronic device calculates the distance between the features of the input face image and the features of the output face image, and the parameters of the neural network can be changed to minimize the distance.

[0025] When the aforementioned instruction is executed by the processor, the electronic device calculates the cosine similarity between the features of the input face image and the features of the output face image, and the parameters of the neural network can be changed to maximize the cosine similarity.

[0026] When the aforementioned instruction is performed by the processor, the electronic device can modify the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold. [Effects of the Invention]

[0027] According to the present invention, it is possible to provide a technique for training a neural network to strike a good balance between the degree of randomization of the input face image and the degree of restoration to the input face image. [Brief explanation of the drawing]

[0028] [Figure 1] An example of an electronic device that randomly reads facial images according to one embodiment is shown. [Figure 2] Figure 1 shows a schematic block diagram of the electronic device. [Figure 3] This figure illustrates a method for training a neural network via a backpropagation purification method according to one embodiment. [Figure 4] An example flowchart of a neural network training method according to one embodiment is shown. [Modes for carrying out the invention]

[0029] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and can be modified in various ways. Therefore, the embodiments are not limited to the specific disclosure, and the scope of this specification includes modifications, equivalents, or substitutions that are part of the technical idea.

[0030] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0031] When it is mentioned that one component is "connected" to another, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0032] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0034] The embodiments will be described in detail below with reference to the attached drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the reference numerals used in the drawings, and redundant explanations will be omitted.

[0035] In this specification, "module" means hardware capable of performing the functions and operations described herein, and may also mean computer program code that performs specific functions and operations, or an electronic recording medium on which computer program code that performs specific functions and operations is mounted, such as a processor or microprocessor.

[0036] In other words, a module means a functional and / or structural combination of hardware for carrying out the technical concept of the present invention and / or software for driving said hardware.

[0037] Figure 1 shows an example of an electronic device that randomizes facial images according to one embodiment.

[0038] Referring to Figure 1, the electronic device 100 can be trained to train a neural network (or a neural network model). Furthermore, the electronic device 100 can perform inference (for example, randomizing face images) using the trained neural network.

[0039] A neural network (or artificial neural network) can include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network generally refers to a model in which artificial neurons (nodes) formed by combinations of synapses change the strength of their synaptic connections through learning, thereby possessing problem-solving capabilities.

[0040] The neurons in a neural network can include combinations of weights or biases. A neural network may include one or more layers, each consisting of one or more neurons or nodes. By changing the weights of the neurons through learning, a neural network can infer the outcome it attempts to predict from any given input.

[0041] Neural networks can include deep neural networks. Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), and LSM (Liquid This may include State Machines, ELMs (Extreme Learning Machines), ESNs (Echo State Networks), DRNs (Deep Residual Networks), DNCs (Differentiable Neural Computers), NTMs (Neural Turning Machines), CNs (Capsule Networks), KNs (Kohonen Networks), and ANs (Attention Networks).

[0042] The electronic device 100 can be implemented on an embedded system using limited hardware resources by employing a lightweight neural network model. The neural network learning device 10 can perform all learning and inference on-device.

[0043] The electronic device 100 may be implemented as a printed circuit board (PCB) such as a motherboard, an integrated circuit (IC), or a system on a chip (SoC). For example, the electronic device 100 may be implemented as an application processor.

[0044] Furthermore, the electronic device 100 may be implemented within a PC (personal computer), a data server, or a portable device.

[0045] Portable devices may include laptop computers, mobile phones, smartphones, tablet PCs, mobile internet devices (MIDs), PDAs (personal digital assistants), EDAs (enterprise digital assistants), digital still cameras, digital video cameras, PMPs (portable multimedia players), PNDs (personal navigation devices or portable navigation devices), handheld game consoles, e-books, or smart devices. Smart devices may include smart watches, smart bands, or smart rings.

[0046] The electronic device 100 can train a neural network by processing the parameters (or weights) of the neural network model. The electronic device 100 can also process the parameters of the neural network model trained with maximum precision to generate a lightweight neural network model.

[0047] The electronic device 100 can obtain new parameters by processing parameters that change during the training of the neural network model, and can retrain the neural network model based on these new parameters.

[0048] The electronic device 100 can acquire an output facial image in which the input facial image has been obfuscated based on the input facial image.

[0049] The electronic device 100 can extract features of the input face image from the output face image to reconstruct the identity information contained in the input face image, based on the input face image. The electronic device 100 can also extract features of the output face image that correspond to the features of the input face image, based on the output face image.

[0050] The electronic device 100 can train a neural network based on the difference between the features of the input face image and the features of the output face image. The electronic device 100 can train a neural network via a backpropagation refinement scheme. This will be explained in detail with reference to Figures 2 and 3.

[0051] The electronic device 100 includes a processor 110 and a memory 120.

[0052] The processor 110 processes data stored in memory 120. The processor 110 can execute computer-readable code (e.g., software) and instructions induced by the processor 110 that are stored in memory 120.

[0053] The processor 110 may be a data processing device embodied in hardware having a circuit with a physical structure for performing a desired operation. For example, the desired operation may include code or instructions included in a program.

[0054] For example, a data processing device embodied in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0055] Memory 120 stores the neural network model or the parameters of the neural network model. Memory 120 also stores instructions (or programs) that can be executed by the processor. For example, the instructions may include instructions for executing the operations of the processor and / or the operations of each component of the processor.

[0056] Memory 120 can be implemented using either a volatile memory device or a non-volatile memory device.

[0057] The volatile memory device may be implemented using DRAM (dynamic random access memory), SRAM (static random access memory), T-RAM (thyristor RAM), Z-RAM (zero capacitor RAM), or TTRAM (Twin Transistor RAM).

[0058] Non-volatile memory devices may be implemented using EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), FeRAM (Ferroelectric RAM), PRAM (Phase Change RAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), Holographic memory, Molecular Eelectronic Memory Device, or Insulator Resistance Change Memory.

[0059] The processor 110 executes code and / or instructions stored in the memory 120, thereby causing one or more operations to be performed by the electronic device 100. The operations performed by the electronic device 100 will be described in detail below with reference to Figures 2 to 4.

[0060] Figure 2 shows a schematic block diagram of the electronic device shown in Figure 1.

[0061] Referring to Figure 2, the electronic device 100 includes an obfuscation module 210 and a feature extractor 230. The obfuscation module 210 and the feature extractor 230 may be implemented by different neural networks or by a single neural network. For the sake of explanation, it will be assumed below that the obfuscation module 210 and the feature extractor 230 are implemented by different neural networks.

[0062] The randomization module 210 generates an output face image based on the input face image, which has been randomized. The randomization module 210 can generate an output face image by sequentially or in parallel performing various transformations on the input face image (e.g., averaging transformation, warping, noise addition, and / or color value adjustment). The various transformations are performed on different layers of the randomization module 210, and the degree of transformation can be determined by the layer parameters. The layer parameters relate to the randomization of the input face image.

[0063] Generally, various transformations are performed in such a way that the output face image is not easily recognizable by humans, and the degree of transformation is set to a high value as a default. When the degree of transformation is set to a high value, the degree of randomization increases, but it becomes difficult to reconstruct the input face image from the output face image. Therefore, it is necessary to set the layer parameters of the randomization module 210 to balance the degree of randomization of the output face image and the degree of reconstruction to the input face image.

[0064] To reconstruct the input face image from the output face image, the difference (or similarity) between the features of the input face image and the features of the output face image may be used. Reconstructing the input face image from the output face image includes reconstructing the identity information contained in the input face image from the output face image. The smaller the difference between the features of the input face image and the features of the output face image, the easier it is to reconstruct the identity information contained in the input face image from the thrust face image. For example, when transforming the input face image via the randomized image processing module 210, the smaller the difference between the features of the input face image and the features of the transformed image (e.g., the output face image), the higher the reconstructive power (e.g., the degree to which identity information contained in the input face image is reconstructed from the output face image). Therefore, the features of the input face image and the features of the output face image can be extracted via the feature extractor 230, and the parameters of the layers of the randomized image processing module 210 can be trained so that the difference between the two features becomes small.

[0065] The feature extractor 230 can extract features of a face image based on the face image (e.g., input face image and / or output face image). For example, the feature extractor 230 may extract features of the input face image to reconstruct identity information contained in the input face image from the output face image. The feature extractor 230 may also extract features of the output face image that correspond to features of the input face image based on the output face image.

[0066] The features of the face image extracted by the feature extractor 230 can be determined by the information to be reconstructed from the input face image (for example, the identity information of the input face image). For example, if the goal is to reconstruct the gender information of the input face image from the output face image, features related to the gender information of the input face image may be extracted via the feature extractor 230.

[0067] By training the layer parameters of the random reading module 210 based on the difference between the features of the input face image and the features of the output face image, the reconstructive power (or degree of reconstruction) from the output face image generated via the random reading module 210 back to the input face image is increased.

[0068] The electronic device 100 can update the parameters of the layers of the randomization module 210 (for example, layers that perform various transformations) so that the degree of randomization and the degree of restoration are trade-offs. This is important because the stronger the degree of randomization, the weaker the degree of restoration, and vice versa, so the degree of randomization and the degree of restoration are trade-offs. The updating of the layer parameters of the randomization module 210 is performed through learning of a backpropagation purification method, which will be explained in detail with reference to Figure 3.

[0069] Figure 3 illustrates a method for training a neural network via a backpropagation purification method according to one embodiment.

[0070] Referring to Figure 3, the electronic device 100 includes a parameter initialization module 310, a random reading module 210, and a feature extractor 230.

[0071] The random reading module 210 processes the input face image I in Various transformations (e.g., averaging, warping, noise addition, and / or color value adjustment) can be performed on the image. These various transformations are carried out by multiple layers included in the randomization module 210, and the following describes the transformations performed by each layer.

[0072] The random reading module 210 includes an averaging layer 320, a warping layer 330, a noising layer 340, and a scaling layer 350.

[0073] Average layer 320 is input face image I inAn averaging transformation (or average transformation) can be performed. The averaging transformation is a transformation that removes details from an image by removing high-frequency information from the input face image while retaining low-frequency information. The averaging transformation includes a mosaic transformation f1 and a transformation that corrects pixels included in one axis of the image to the average of those pixels (e.g., a horizontal mean transformation f2 and / or a vertical mean transformation f3).

[0074] Mosaic transformation f1 is performed on the input face image I in This transformation divides the image into multiple blocks and corrects the pixel values ​​of each block to the average of the pixel values ​​of the multiple blocks. For example, the averaging layer 320 divides the image into M x N blocks and calculates the average of the pixel values ​​of the blocks. The averaging layer 320 can correct the pixel values ​​of each block to the average of the pixel values ​​of the blocks. As a result, all pixel values ​​within the M x N blocks become identical, thereby removing high-frequency information from the input face image. The degree of mosaic transformation f1 is determined based on the parameter θ1 of the averaging layer 320. For example, the larger the parameter θ1, the higher the degree of mosaic transformation, and the output face image I out The degree of indiscriminate reading increases.

[0075] Horizontal average transformation f2 is performed on the input face image I in This is a transformation that corrects the pixel values ​​of blocks included in the horizontal axis direction to the average of the pixel values ​​of the blocks. The horizontal average transformation f2 is determined based on the parameter θ2 of the average layer 320. For example, the larger the parameter θ2, the higher the degree of the horizontal average transformation f2, and the output face image I out The degree of indiscriminate reading increases.

[0076] The vertical average transformation f3 is performed on the input face image I inIt is a conversion that corrects the pixel values of the blocks included in the vertical axis direction to the average of the pixel values of the blocks. The vertical average conversion f3 is determined based on the parameter θ3 of the average layer 320. For example, the larger the parameter θ3, the higher the degree of the vertical average conversion f3, and the higher the degree of scrambling of the output face image I out becomes.

[0077] The average layer 320 can combine the input face images on which the mosaic conversion f1, the horizontal average conversion f2, and the vertical average conversion f3 have been performed. For example, the average layer 320 can normalize the blocks of the input face image on which the mosaic conversion f1, the horizontal average conversion f2, and the vertical average conversion f3 have been performed through the softmax operation. The average layer 320 can combine the normalized blocks to generate one overlapping block. The image on which the average conversion has been performed may be composed of a plurality of overlapping blocks. Specifically, the average layer 320 can combine the input face images on which the mosaic conversion f1, the horizontal average conversion f2, and the vertical average conversion f3 have been performed through the following formula (1).

[0078]

Equation

[0079] In Equation (1), B1 represents the block of the image on which the mosaic conversion f1 has been performed, B2 represents the block of the image on which the horizontal average conversion f₂ has been performed, and B3 represents the block of the image on which the vertical average conversion f3 has been performed. JPEG2026049693000003.jpg17167B avg represents the overlapping block, and c represents the RGB channels of the input face image.

[0080] The warping layer 330 warps the input face image that has undergone the average transformation (hereinafter referred to as the warping transformation f4) and rearranges the pixels of the input face image that has undergone the average transformation. That is, the warping transformation f4 can include deformation of the geometric properties of the image. For example, the warping layer 330 can distort the face image based on grid points (e.g., intersections between blocks) within an M × N block of the face image (e.g., the input face image that has undergone the average transformation by the averaging layer 320). The warping layer 330 can move each grid point by a value determined by the parameter θ4 of the warping layer 330. The warping layer 330 can move the grid points by (θ4 × Δ), where Δ represents the size of the block (e.g., the vertical size of the block when moving the grid points vertically) (e.g., the horizontal size of the block when moving the grid points horizontally). The warping layer 330 moves the grid points so that they do not overlap. Therefore, the warping transform f4 causes distortion in the face image. The larger the parameter θ4, the greater the degree of the warping transform f4, as the grid points are moved more, resulting in a distorted output face image I out The degree of indiscriminate reading increases.

[0081] The noising layer 340 may add noise to the input face image whose pixels have been rearranged (for example, the input face image transformed by the averaging layer 320 and the warping layer 330). The noising layer 340 can increase the complexity of the image by adding high-frequency components to the face image through noise addition, thereby increasing the reconstructive power from the transformed image back to the original image (for example, the input face image).

[0082] The noise may include sinusoidal noise, checkerboard noise, and / or speckle noise. For example, the noising layer 340 may perform a sinusoidal noise addition transformation f5 on the face image. The noising layer 340 may perform a checkerboard noise addition transformation f6 on the face image. The noising layer 340 may perform a speckle noise addition transformation f7 on the face image.

[0083] The sinusoidal-based noise addition transformation f5 can be performed based on the parameter θ5 of the noising layer 340. For example, the noising layer 340 generates sinusoidal-based noise along the axis with parameter θ5 for each block of the face image (e.g., the input face image transformed by the averaging layer 320 and the warping layer 330). The noising layer 340 can add sinusoidal-based noise (e.g., a periodic pattern to the face image) to the face image, thereby enhancing the high-frequency components.

[0084] The noise addition transformation f6 of the checkerboard substrate can be performed based on the parameter θ6 of the noising layer 340. For example, the noising layer 340 may add an N×N (e.g., 4×4) checkerboard pattern to each block of the face image. The N×N checkerboard pattern may be adjusted by the parameter θ6. By adding the N×N checkerboard pattern, high-frequency components can be added to the face image.

[0085] The speckle-based noise addition transformation f7 can be performed based on the parameter θ7 of the noising layer 340. For example, the noising layer 340 may assign parameter θ7 to the central block of each block in the face image. The noising layer 340 can then determine the pixel values ​​of the remaining blocks by bilinear interpolation from the central block to which parameter θ7 is assigned.

[0086] The noising layer 340 can combine face images that have undergone noise enhancement transformations f5 for sinusoidal substrates, f6 for checkerboard substrates, and f7 for speckle substrates. For example, the noising layer 340 may normalize blocks of input face images that have undergone noise enhancement transformations f5 for sinusoidal substrates, f6 for checkerboard substrates, and f7 for speckle substrates via a softmax operation. The noising layer 340 can combine the normalized blocks with blocks of images that have undergone a warping transformation f4 by the warping layer 330 to generate a single superimposed block. Images that have undergone noise enhancement transformations may consist of multiple superimposed blocks. Specifically, the noising layer 340 can combine input face images that have undergone noise enhancement transformations f5 for sinusoidal substrates, f6 for checkerboard substrates, and f7 for speckle substrates via the following formula (2).

[0087]

number

[0088] In equation (2), B warped B1 shows the image block after warping transformation f4 has been applied, B5 shows the image block after noise addition transformation f5 has been applied to the sinusoidal substrate, B6 shows the image block after noise addition transformation f6 has been applied to the checkerboard substrate, and B7 shows the image block after noise addition transformation f7 has been applied to the speckle substrate. JPEG2026049693000005.jpg17167B noi 'c' indicates overlapping blocks, and 'c' indicates the RGB channels of the input face image.

[0089] Scaling layer 350 adjusts the color values ​​of the noisy face image (e.g., the input face image transformed by the averaging layer 320, warping layer 330, and noising layer 340) (hereinafter referred to as scaling transformation f8) to generate the output face image. Scaling transformation f8 is a transformation that adjusts the color values ​​(e.g., color intensity) of the face image.

[0090] The scaling transformation f8 can be performed based on the parameter θ8 of the scaling layer 350. For example, the scaling layer 350 may adjust the color values ​​of each block of the face image by parameter θ8. If parameter θ8 is greater than 1, the brightness of the block increases, and if parameter θ8 is less than 1, the brightness of the block decreases. By adjusting the color intensity of the image, the scaling transformation f8 increases the degree of reconstructiveness (e.g., the degree of reconstruction from the transformed face image (e.g., output face image) to the original face image (e.g., input face image)) while also increasing the degree of random reading.

[0091] The feature extractor 230 includes multiple feature extractors 360 and 370. Feature extractor 360 outputs a face image I out Features are extracted from the input face image I. Feature extractor 370 extracts features from the input face image I. in Features are extracted. The configuration and operation of feature extractors 360 and 370 are substantially the same as those of feature extractor 230, and redundant explanations are omitted.

[0092] In summary, the input face image I is processed through multiple layers included in the random reading module 210. in Convert and output face image I out The method for obtaining (or generating) the input face image I was described in detail. in The conversion is performed based on the parameters of multiple layers (e.g., parameters θ1 to θ8) included in the random reading module 210. That is, the input face image I is converted via the update (or learning) of parameters θ1 to θ8. in By converting, the output face image Iout The degree of random reading and the ability to reconstruct the results can be determined. Below, the output face image I out This paper details a method for training a neural network via backpropagation to determine the parameters θ1~θ8, which trade off the degree of random reading and the reconstructive power (e.g., the degree of identity information reconstruction from the output face image to the input face image).

[0093] First, let's explain how the backpropagation refinement method differs from conventional backpropagation methods. Conventional backpropagation methods calculate a loss function by comparing the output generated via the forward propagation process with the measured value (ground truth), and update the neural network parameters via the backpropagation process. On the other hand, the backpropagation refinement method updates the neural network parameters by repeatedly executing the forward propagation process and the backpropagation process (for example, different from calculating the loss function by comparing with the measured value). The forward propagation process includes operations performed by the randomization module 210 (for example, acquiring an output face image based on an input face image) and the feature extractor 230 (for example, extracting features from the input face image and extracting features from the output face image). The backpropagation process may include operations to update the neural network parameters.

[0094] The following describes in detail a method for updating the parameters of a neural network via a backpropagation purification method in the present invention.

[0095] The parameters of the randomized reading module 210 (for example, parameters θ1 to θ8) control the transformation of the input face image, and the quality of the output face image can be determined based on how these parameters are initialized.

[0096] Parameters are classified into four categories according to their characteristics. The parameter initialization module 310 can initialize parameters of different categories using different initialization methods.

[0097] Parameters may include fixed parameters, uniform parameters, color parameters, and composing parameters, depending on their characteristics. The characteristics and initialization methods of each parameter are described below.

[0098] Fixed parameters refer to parameters that are fixed at a specific value. Fixed parameters are used to remove high-frequency information (or facial detail) from a face image, which can increase the degree of randomization (or human indecipherability, HI). Parameters θ1~θ3 and θ6 may be fixed parameters because they are used to remove high-frequency information from the face image. Parameters θ1~θ3 and θ6 may be fixed at a specific value (e.g., 1). For example, if mosaic transformation f1 is performed based on parameter θ1 (e.g., 1), high-frequency information from the input face image is removed by correcting all pixels in each block to the same average value.

[0099] In the case of fixed parameters, since they always have a fixed value, there is no need to perform separate initialization by the parameter initialization module 310.

[0100] A uniform parameter is a parameter whose values ​​are uniformly distributed within a specific range. Parameters θ4~θ5 and θ7 are uniform parameters.

[0101] JPEG2026049693000006.jpg23167 For example, the parameter initialization module 310 may set the range in which the grid points move according to the parameter θ4 in the warping transform f4 to [-0.3, 0.3]. The parameter initialization module 310 may also set the initial value of the parameter θ4 (e.g., 10 / 6). By moving the grid points by values ​​uniformly distributed in [-0.3, 0.3] (e.g., 10 / 6), sufficient distortion is caused in the face image without the grid being greatly deformed. The noise addition transform f5 for the sinusoidal substrate and the noise addition transform f7 for the speckle substrate are initialized in a similar manner to the warping transform f4, so redundant explanations are omitted.

[0102] The color parameter controls the color conversion and is used to adjust the color of the input face image. The parameter θ8 is used to adjust the color intensity of the image and may also be a color parameter. When parameter θ8 is greater than 1, the brightness of the input face image increases, and when parameter θ8 is less than 1, the brightness of the input face image decreases.

[0103] The parameter initialization module 310 can determine brightness increase and brightness decrease with equal probability (e.g., 50%) and set the value of parameter θ8. The parameter initialization module 310 can set parameter θ8 within a specific range (e.g., 1 / θmin ~θ max It can be initialized within ). Therefore, color conversion can preserve the high-frequency information of the image while enabling image variance.

[0104] Combination parameters are parameters used to combine the results of various transformations. Combination parameters may also be used to combine transformed image blocks. JPEG2026049693000007.jpg17167 For example, the combination parameters can be normalized via softmax calculation to appropriately adjust the ratio of the combined blocks.

[0105] The parameter initialization module 310 can uniformly initialize combination parameters within a specific range (for example, 0 to 1).

[0106] The parameters of the neural network (e.g., the randomization module 210) initialized via the parameter initialization module 310 may be learned via the following backpropagation refinement scheme and determined to be values ​​that trade off the degree of randomization of the output face image with the degree of identity information recovery from the input face image to the output face image. The following describes in detail how to update (or optimize) the initialized parameters.

[0107] Output face image I generated based on the initial parameter values ​​of JPEG2026049693000008.jpg35167 out Even if the degree of indiscriminate reading is high, the reconstructive power (for example, output facial image I) out Input face image I in The degree of identity information recovery is low. For example, the output face image I out Features (for example, extracted by feature extractor 360) and input face image I in The features (for example, those extracted by the feature extractor 370) may differ significantly.

[0108] The parameters of the randomized reading module 210 are learned through four types of loss functions.

[0109] Fixed parameters (for example, parameters θ1 to θ3 and θ6) may be excluded from the update process as they are fixed values.

[0110] The electronic device 100 can modify the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold (e.g., margin). Specifically, the uniformity parameter and color parameter may be learned by the electronic device 100 as follows.

[0111] The uniform parameters (e.g., parameters θ4-θ5 and θ7) may be learned via a first loss function. The first loss function updates the uniform parameters in a direction that increases the degree of randomization. The first loss function is expressed as shown in equation (3) below.

[0112]

number

[0113] In equation (3), θ i λ represents the uniform parameter, i is the parameter θ i This shows the margin set for the given value (for example, 0.05 for θ4, 0 for θ5, and 0.1 for θ7).

[0114] The first loss function may be set such that the degree of random reading increases as the uniformity parameter approaches the margin.

[0115] The color parameter (e.g., parameter θ8) is learned via a second loss function. The second loss function can update the color parameter in a way that increases the degree of random reading. The second loss function is expressed as shown in equation (4) below.

[0116]

number

[0117] In equation (3), [ ] indicates the indicator function, and θ i indicates the color parameter, λ i is the parameter θ i This shows the margin set for that value (for example, 1.05 for θ8).

[0118] The second loss function can determine whether the color parameter is optimized in the direction of increasing or decreasing brightness.

[0119] JPEG2026049693000011.jpg34167

[0120] The electronic device 100 can calculate the distance (e.g., Euclidean distance) between the features of the input face image and the features of the output face image. The electronic device 100 calculates a third loss function based on the Euclidean distance. The third loss function is calculated based on the output face image I out Features and input face image I in The parameters are set to minimize the Euclidean distance between the features. The electronic device 100 can change the parameters of the neural network so that the distance is minimized. Specifically, the third loss function is expressed as shown in equation (5) below.

[0121]

number

[0122] JPEG2026049693000013.jpg18167

[0123] The electronic device 100 can calculate the cosine similarity between the features of the input face image and the features of the output face image. The electronic device 100 calculates a fourth loss function based on the cosine similarity. The fourth loss function is calculated based on the output face image I out Features and input face image I in It is set to maximize the cosine similarity between the features. The electronic device 100 can change the parameters of the neural network so that the cosine similarity is maximized. Specifically, the fourth loss function is expressed as shown in equation (6) below.

[0124]

number

[0125] JPEG2026049693000015.jpg18167

[0126] In conclusion, the first to fourth loss functions can be learned end-to-end via the loss function shown in equation (7).

[0127]

number

[0128] The loss function in equation (7) is a weighted sum of the first to fourth loss functions, and was set up to find the optimal parameter that balances the degree of random reading and the restorative force.

[0129] The parameters can be learned so that the loss function obtained by the forward propagation process is optimized via the backpropagation process. Specifically, the electronic device 100 can calculate the derivative of the loss function and calculate the gradient for each parameter. The derivative of the loss function is expressed as shown in the following equations (8) to (10).

[0130]

number

[0131]

number

[0132]

number

[0133] The parameters in equations (8) to (10) are the same as those in equations (1) to (7).

[0134] The electronic device 100 can update parameters based on gradients. This process (for example, calculating a loss function via a forward propagation process and then calculating the derivative of the loss function via a backpropagation process to update parameters) is repeated until convergence is achieved, determining the parameters of a neural network (e.g., a randomization module 210) where the degree of randomization of the output face image and the reconstructive power (e.g., the degree of identity information reconstruction from the output face image) are traded off.

[0135] Learning through backpropagation allows the machine learning algorithm to recognize the output face image even after the parameters have been initialized, while simultaneously making it difficult for humans to easily recognize the output face image.

[0136] Figure 4 shows an example of a flowchart for a neural network learning method according to one embodiment.

[0137] Referring to Figure 4, operations 410 to 470 are performed sequentially, but are not limited to this order. For example, two or more operations may be performed in parallel. Operations 410 to 470 are substantially the same as the operations of the electronic device described with reference to Figures 1 to 3 (for example, the electronic device 100 in Figure 1). A detailed explanation is omitted here.

[0138] In operation 410, the electronic device 100 obtains an output face image in which the input face image has been distorted based on the input face image.

[0139] In operation 430, the electronic device 100 extracts features from the input face image to reconstruct the identity information contained in the input face image from the output face image based on the input face image.

[0140] In operation 450, the electronic device 100 extracts features of the output face image that correspond to the features of the input face image based on the output face image.

[0141] In operation 470, the electronic device 100 trains a neural network based on the difference between the features of the input face image and the features of the output face image.

[0142] The embodiments described above can be embodied in hardware components, software components, and / or combinations of hardware and software components. For example, the adaptive supersampling apparatus, method, and components described in the embodiments can be embodied using a general-purpose computer or a special-purpose computer, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPGA (Field Programmable Gate Array), PLU (Programmable Logic Unit), microprocessor, or other adaptive supersampling apparatus capable of executing and responding to commands. The processing apparatus can execute an operating system (OS) and software applications that run on the OS. The processing adaptive supersampling apparatus may also access, store, manipulate, process, and generate data in response to the execution of software. For convenience of understanding, the processing adaptive supersampling apparatus has sometimes been described as using only one, but a person with ordinary skill in the art will see that the processing adaptive supersampling apparatus may include multiple processing elements and / or multiple types of processing elements. For example, the processing adaptive supersampling apparatus may include multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.

[0143] The software may include computer programs, code, instructions, or any combination thereof, and may configure or instruct a processing adaptive supersampling device as desired, independently or collectively. The software and / or data may be permanently embodied in any type of machine, component, physical adaptive supersampling device, virtual adaptive supersampling device, computer storage medium, or adaptive supersampling device, or transmitted signal waves, in order to be interpreted by or to provide instructions or data to the processing adaptive supersampling device. The software may be distributed on a networked computer system and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0144] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0145] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described technique may be performed in a different order than described, and / or the described system, structure, apparatus, circuit, and other components may be combined or assembled in a different manner than described, or replaced or substituted by other components or equivalents, and still achieve the desired results.

Claims

1. A method for training a neural network to randomize facial images, An operation to obtain an output face image that has been randomly processed from the input face image based on the input face image, An operation to extract features of the input face image in order to reconstruct identity information contained in the input face image from the output face image based on the input face image, An operation to extract features of the output face image that correspond to the features of the input face image based on the output face image, The operation involves training the neural network based on the difference between the features of the input face image and the features of the output face image, Methods that include...

2. The method according to claim 1, wherein the operation to acquire the output face image includes the operation to input the input face image to the neural network and generate the output face image.

3. The operation to generate the aforementioned output face image is: The operation of performing an average transformation on the aforementioned input face image, The operation involves warping the input face image after the average transformation has been performed, and rearranging the pixels of the input face image after the average transformation has been performed. The operation of adding noise to the input face image in which the aforementioned pixels have been rearranged, The operation involves adjusting the color values ​​of the input face image to which the noise has been added, and generating the output face image. The method according to claim 2, including the method described in claim 2.

4. The method according to claim 3, wherein the averaging transformation includes a mosaic transformation and a transformation that corrects pixels included in one axis of the image to the average of the pixels.

5. The method according to claim 3, wherein the noise includes noise from a sinusoid substrate, noise from a checkerboard substrate, and noise from a speckle substrate.

6. The operation of training the neural network includes updating the parameters of the neural network via a backpropagation refinement scheme based on the difference between the features of the input face image and the features of the output face image. The parameters of the neural network are as described in claim 2, relating to the randomization of the input face image.

7. The aforementioned backpropagation purification method is This method involves repeatedly performing forward and backpropagation processes to determine the parameters of the neural network where there is a trade-off between the degree of randomization of the output face image and the degree of identity information reconstruction from the output face image. The forward propagation process includes the operation of acquiring the output face image, the operation of extracting features from the input face image, and the operation of extracting features from the output face image. The method according to claim 6, wherein the backpropagation process includes updating the parameters of the neural network.

8. The operation to update the parameters of the aforementioned neural network is: The operation of calculating the distance between the features of the input face image and the features of the output face image, The operation of changing the parameters of the neural network so that the aforementioned distance is minimized, The method according to claim 7, including the method described in claim 7.

9. The operation to update the parameters of the aforementioned neural network is: The operation of calculating the cosine similarity between the features of the input face image and the features of the output face image, The operation of changing the parameters of the neural network so that the cosine similarity is maximized, The method according to claim 7, including the method described in claim 7.

10. The method according to claim 7, wherein the operation to update the parameters of the neural network includes the operation to change the parameters of the neural network so that the parameters of the neural network do not exceed a preset threshold.

11. An electronic device that randomly reads facial images, Processor and Memory for storing instructions, Includes, When the aforementioned instruction is performed by the processor, the electronic device, Based on the input face image, an output face image is obtained by randomly processing the input face image via a neural network. An electronic device wherein the neural network is trained via the method described in any one of claims 1 to 10.

12. An electronic device for training a neural network that randomizes facial images, Processor and Memory for storing instructions, Includes, When the aforementioned instruction is executed by the processor, the electronic device, Based on the input face image, an output face image is obtained by randomly reading the input face image. Based on the input face image, features of the input face image are extracted from the output face image to reconstruct the identity information contained in the input face image. Based on the output face image, features of the output face image corresponding to the features of the input face image are extracted. An electronic device in which the neural network is trained based on the difference between the features of the input face image and the features of the output face image.

13. The electronic device according to claim 12, wherein when the instruction is performed by the processor, the electronic device inputs the input face image to the neural network and generates the output face image.

14. When the aforementioned instruction is executed by the processor, the electronic device, An average transformation is performed on the aforementioned input face image. The input face image after the average transformation is warped, and the pixels of the input face image after the average transformation are rearranged. Noise is added to the input face image in which the aforementioned pixels have been rearranged. The electronic device according to claim 13, wherein the color values ​​of the input face image to which the noise has been added are adjusted, and the output face image is generated.

15. The electronic device according to claim 14, wherein the averaging transformation includes a mosaic transformation and a transformation that corrects pixels included in one axis of the image to the average of the pixels.

16. When the aforementioned instruction is executed by the processor, the electronic device, Based on the difference between the features of the input face image and the features of the output face image, the parameters of the neural network are updated via a backpropagation purification method. The parameters of the neural network relate to the randomization of the input face image, as described in claim 13.

17. The aforementioned backpropagation purification method is a method that repeatedly performs forward propagation and backpropagation processes in order to determine the parameters of the neural network in which there is a trade-off between the degree of randomization of the output face image and the degree of identity information reconstruction from the output face image. The forward propagation process is a process of acquiring the output face image, extracting the characteristics of the input face image, and extracting the features of the output face image. The electronic device according to claim 16, wherein the backpropagation process is a process for updating the parameters of the neural network.

18. When the aforementioned instruction is executed by the processor, the electronic device, The distance between the features of the input face image and the features of the output face image is calculated. The electronic device according to claim 17, wherein the parameters of the neural network are changed so that the aforementioned distance is minimized.

19. When the aforementioned instruction is executed by the processor, the electronic device, The cosine similarity between the features of the input face image and the features of the output face image is calculated. The electronic device according to claim 17, wherein the parameters of the neural network are changed so as to maximize the cosine similarity.

20. When the aforementioned instruction is executed by the processor, the electronic device, The electronic device according to claim 17, wherein the parameters of the neural network are changed so that the parameters of the neural network do not exceed a preset threshold.

Images