Imaging system

Abstract

To provide an imaging system.SOLUTION: An imaging system (1) includes: a planar imaging device (4) on which a plurality of pixels each composed of a light-sensitive element are arranged; a mask (2) which is arranged on a front face of the imaging device (4) and on which a pattern for modulating incident light is formed; a reading unit (5) for reading an imaged picture of an imaged subject; and a recognition unit (12) for performing sexuality determination on additional information on the subject from the read imaged picture. The pattern includes a plurality of translucent units (21) arranged for leading beams from a plurality of directions of beams from the subject to one pixel. Thereby, individual recognition is disabled while enabling recognition of additional information on an individual for predetermined use to perform privacy protection.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a camera technology capable of protecting privacy by capturing an image of a subject by modulating the image to a level that makes the subject unrecognizable as an individual. [Background technology]

[0002] Traditionally, cameras project a retinal image, i.e., a human-understandable collected light image, onto an image sensor through a lens, i.e., measure the light receiving brightness at each pixel of the image sensor, thereby digitizing and capturing optical images. The captured image data is generally read out in a raster scan order while maintaining the spatial relationship, and is then transferred, for example, via an Internet line and saved as a data file. If information is hacked or leaked during transfer or storage, the contents can be easily observed. Today, such data leaks and unilateral disclosure by third parties have led to serious image privacy issues. For example, there have been cases in which a glasses-type IoT device equipped with a camera was banned from use in restaurants and its release was canceled, and cases in which a third party requested the deletion of images uploaded to SNS.

[0003] In recent years, lensless cameras or flat cameras have been proposed (for example, Patent Document 1). This type of camera has a plate-shaped modulator that modulates transmitted light instead of a lens, which allows the imaging device to be made thinner. The imaging device includes a modulator that modulates light intensity using a first pattern formed in a concentric circle shape, an image sensor that converts the light image transmitted through the modulator into image data, and an image processing unit that performs cross-correlation calculation between the image data output from the image sensor and pattern data indicating the second pattern, thereby enabling the restoration of the subject image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-61109 A Summary of the Invention [Problem to be solved by the invention]

[0005] Surveillance cameras and the like are used to identify individuals for crime prevention purposes, but many of today's smartphones, smart speakers, and IoT devices are not necessarily intended to identify individuals or record evidence, and are instead used as sensors and monitors for recognizing individuals' facial expressions, gestures, and behavior. Drones and cameras for automatic driving are also sensors for environmental recognition and obstacle detection, and do not need to record the privacy of people captured there. In this way, while there are uses of cameras that are not originally intended to record or store privacy information, the use of cameras itself is a cause for concern about privacy violations, limiting their use, and cameras cannot be used simply as sensors for gesture recognition, creating a dilemma in which applications and services for the coming IoT and Society 5.0 era cannot be started. As a general method for solving this problem, it has been proposed to encode and transfer an image on the edge side after capturing it, and then decode it on the server side before recognition. However, even with this method, the concern that the captured image before encoding and the decoded image may be leaked due to hacking or information leakage cannot be completely eliminated.

[0006] Furthermore, in the imaging device described in Patent Document 1, the data acquired by the image sensor is image information that can be restored, and therefore there is a risk that the information may be hacked or leaked by a third party and made public, and therefore there is no consideration for privacy protection.

[0007] The present invention has been made in view of the above, and has an object to provide a camera and imaging system that enable recognition (identification) of attached information of an individual while protecting the privacy of the individual subject. [Means for solving the problem]

[0008] The camera of the present invention includes a planar imaging element having an array of multiple pixels made of photosensitive elements, and a planar modulation element arranged in front of the imaging element and having a pattern formed thereon for modulating incident light, the pattern including an array of multiple light-transmitting sections each of which guides light rays from a subject coming from multiple directions to one pixel.

[0009] According to the present invention, the light from the subject is optically modulated by the modulation element and then captured by the imaging element. Although the captured image destroys spatial projection information such as an optical retinal image, it is possible to retain information that is essentially necessary for recognition. This makes it difficult to visually recognize the contents from a recorded or leaked data string, thereby protecting privacy. Effect of the Invention

[0010] According to the present invention, it is possible to take a photograph in which individual recognition of a subject is impossible, while additional information about the individual for a desired purpose is recognized, thereby protecting privacy. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram, partly in cross section, seen from the side, showing the configuration of an imaging system according to the present invention; [Diagram 2] 3 is a diagram showing the relationship between the pattern of a modulation element and pixels of an imaging element. FIG. [Diagram 3] This figure explains the relationship between the presence or absence and type of modulation element and the captured image. (A) is a lensless case, (B) is a case where a pinhole-shaped hole (pinhole) is drilled, and (C) is a case where a mask having multiple or different sized light-transmitting parts is interposed on the surface. [Figure 4] FIG. 13 shows other patterns of a modulation element, in which (A) is a mask in which light-transmitting portions of different sizes are formed, and (B) is a mask in which light-transmitting portions are formed randomly or in a sparsely and densely pattern. [Diagram 5] FIG. 13 shows another embodiment of a modulation element. [Figure 6]Illustrated lensless imaging with small and large distances between the image and the encoding plane. (A) is for small distances, and (B) is for large distances. [Figure 7] FIG. 1 is a system diagram illustrating visual privacy protection for face recognition via lensless imaging. [Figure 8] In the figure showing a visual comparison of different imaging systems, the scale above the measurements and patterns indicates the ratio between blur and exposure, and the dimension of the fixation pattern is m=32×32. [Figure 9] ,Figure showing the confusion matrix of LwoC-woRec for multiple Hi and Ri, where each value of (i,j) indicates the Top1 accuracy (%) of the jth recognition feature Rj on the input lensless measurement of the ith coding pattern Hi. [Figure 10] Figure 10(A) shows the Top1 accuracy (%) of VGG-Face2 for coded image size n=63×63 (Figure 10(A)) and 127×127 (Figure 10(B)) with coded pattern size m=32×32. [Figure 11] Fig. 1 illustrates the learning patterns and measurements of human visual privacy protection with various weights, n = 63 × 63, m = 32 × 32. [Figure 12] This figure shows the confusion matrix between the training pattern Hi and the recognition function Ri in VGG-Face2, with 10 classes, n = 63 × 63, and b = 32 × 32. [Figure 13] FIG. 13 illustrates another embodiment of a hardware implementation for lensless imaging. [Figure 14] FIG. 1 shows the displayed image, the actual coded pattern on the spatial light modulator, and the actual captured measurements (rescaled to maximum and minimum values ​​for better visual quality). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 is a schematic diagram of an imaging system 1 according to the present invention, partially in cross section, viewed from the side. In FIG. 1, the imaging system 1 includes a camera 11 and a recognition unit 12. The camera 11 includes, from the front in the optical axis direction, a mask 2, which is an embodiment of a modulation element, an imaging element 4, a thin bonding layer 3 that optically bonds the mask 2 and the imaging element 4, and a readout unit 5 that reads out image data captured by the imaging element 4 from each pixel. For ease of explanation, the camera 11 is shown in an exaggerated size relative to the subject P. In addition, the bonding layer 3 may be only an adhesive layer in an integrated form, or may further adopt a physical connection structure.

[0013] Camera 11 is a digital camera equipped with an image sensor 4. Image sensor 4 is typically configured with a large number of pixels 42 arranged in a matrix on the front surface of a rectangular, plate-like (planar) main body 41. Each pixel 42 is a tiny photosensitive element such as a CCD, and generates a voltage signal according to the luminance of received light.

[0014] The mask 2 is a sheet-like or thin plate-like body having a size corresponding to the imaging element 4. The mask 2 has light-shielding properties, while light-transmitting sections 21 consisting of a plurality of holes or light-transmitting regions are formed at appropriate positions on the surface. Conversely, the mask 2 may have light-transmitting properties, while light-shielding treatment is applied to the surface other than the light-transmitting sections 21.

[0015] 2 is a diagram showing the arrangement relationship between the pattern (modulation pattern) of the light-transmitting portions 21 of the mask 2 and the pixels 42 of the image sensor 4. The light-transmitting portions 21 are preferably provided corresponding to the pixels 42, and are formed at a predetermined pitch in at least one row and column direction of the pixels 42.

[0016] The size of the light-transmitting portion 21 does not need to be uniform, and it is preferable that all or a part of the light-transmitting portion 21 is equal to or larger than the size of the pixel 42. In FIG. 1 and FIG. 2, the size of the light-transmitting portion 21 is several times the size of the pixel 42, but may be several tens to several hundreds of times. By including the light-transmitting portion 21 of the mask 2 having the above-mentioned large size, the light beams L1, L2 (or other light beams) incident from a plurality of directions on the subject P are made to enter the same pixel 42. In this way, the light beams from a plurality of directions are made to enter the same pixel 42, that is, modulated, that is, the spatial projection information is optically destroyed without forming a retinal image, and the image is captured, thereby reducing the level at which the subject P cannot be recognized as a person from the captured image itself.

[0017] FIG. 3 is a diagram for explaining the relationship between the presence or absence and type of mask 2 and the captured image. FIG. 3(A) is a lensless system in which an image of a subject is captured by an image sensor 4 without the interposition of a mask 20A. In FIG. 3(A), light rays from each direction of the subject are evenly incident on all pixels 42, so the captured image is uniform over the entire surface and is completely meaningless. On the other hand, as shown in FIG. 3(B), when only one pinhole-shaped hole (pinhole: pinhole) is drilled in the mask 20B, only incident light from one direction of the subject enters through the pinhole and measures different luminance, based on the principle of a pinhole camera, so that the image is completely formed like a retinal image like a normal photograph.

[0018] FIG. 3(C) shows an arrangement in which a mask 2a having multiple or different sized light-transmitting portions 21a on its surface is interposed, so that light rays transmitted through multiple light-transmitting portions 21a are combined and directed to each of pixel 421 and pixel 422, and light rays transmitted through the same light-transmitting portion 21a are combined and directed to both pixel 421 and pixel 422, thereby capturing an image.

[0019] In the above, in the case of Figure 3(A), since all information is visually lost, privacy protection is possible, but since no information remains, that is, all pixels are integrated (averaged) in the same way and cannot be separated, it becomes impossible to distinguish what is shown as the subject image, such as image recognition. In the case of Figure 3(B), on the other hand, since the subject image itself is shown, there is no loss of data, and it is actually more vulnerable to privacy.

[0020] On the other hand, as in the case of Fig. 3(C), if the captured image is put into this intermediate state using a mask 2a, it is possible to make it visually impossible to know what is being captured in the captured image. Therefore, in this case, even if the captured image itself is hacked or leaked, the meaningless state is maintained, and even if the captured image and the mask information are stolen and image processing is performed for reproduction, reproduction to the level of personal recognition is not possible, so privacy protection is still ensured. For example, when the captured image is put into the intermediate state as in Fig. 3(C), as can be seen from the captured image G, even if the shading pattern is measured and auxiliary information such as the position information of the subject can be recognized, the subject itself cannot be reproduced.

[0021] Fig. 4 and Fig. 5 are diagrams showing other aspects of the modulation element, Fig. 4 showing another mask pattern, and Fig. 5 showing another embodiment. Fig. 4(A) shows a mask 2b in which light transmitting portions 21b and 22b of different sizes are formed, and Fig. 4(B) shows a mask 2c in which light transmitting portions 21c are formed randomly or coarsely and densely. The shape of the light transmitting portions may be any of a square (including a slit shape), a polygon, and a circle.

[0022] FIG. 5 shows a plate-like body 2d of a thin layer having light transmission, instead of a mask 2 as an example of a modulation element. The plate-like body 2d may be in the form of a sheet. At least one of the front and back surfaces of the plate-like body 2d is formed into a rough surface 21d having a convex and concave shape (corresponding to a light-transmitting portion). The convex and concave shape of the rough surface 21d may include a minute convex lens shape or a concave lens shape. The size in the surface direction of the convex and concave surfaces forming the rough surface 21d may be a size corresponding to the size of the pixel 42 or several to a hundred times the size. The convex and concave surfaces forming the rough surface 21d correspond to the light-transmitting portion.

[0023] The plate-like body 2d is not a condensing lens that enables regular light collection, but refracts light rays, for example, L11, L12, and L13 from multiple directions, within the plate-like body 2d and directs them in irregular directions, as shown in Fig. 5. In other words, the rough surface 21d causes the transmitted light rays L11, L12, and L13 to be incident on pixels 42 that are in a non-corresponding positional relationship, such as incident on the same pixel 42 or on a different pixel 42 that is skipped over. As a result, the spatial projection information of the image from the subject is optically destroyed, and the captured image becomes meaningless information that cannot be used to recognize individuals.

[0024] Returning to FIG. 1, the readout unit 5 outputs a voltage signal (measurement signal) generated by each pixel 42 of the image sensor 4. The readout unit 5 reads out the signals of each pixel 42 in a predetermined order along the array direction, for example, in sequence in correspondence with raster scanning. In addition, the readout unit 5 may read out the signals from the image sensor 4 in a random order or by adding and reading out the signals of a plurality of pixels, and may perform electronic encryption to output an image that is difficult for a person to understand what is being shown, and this image is effectively recognized (determined) by machine learning, for example, by the recognition unit 12 having parameters suitable for the purpose of determining the gender of the subject. The recognition unit 12 may be an integral or semi-integral structure with the camera 11, or may be disposed in a remote location (for example, a monitor room) via a wired or wireless connection, or an internet line.

[0025] The recognition unit 12 performs recognition (judgment) on input image information using parameters acquired through machine learning, and outputs the result. The recognition unit 12 effectively performs recognition (judgment) specialized for a specific intended use. The parameters stored in the parameter storage unit 121 of the recognition unit 12 are modeled through machine learning. As the machine learning, at least one of the so-called supervised learning, unsupervised learning, reinforcement learning, and deep learning learning methods is adopted.

[0026] Machine learning has an input layer, an output layer, and at least one hidden layer between them that simulate (model) a neuron network, and each layer has a structure in which multiple nodes are connected by edges. The parameters refer to the weight values ​​of each edge in each layer. For example, in supervised learning, when recognizing (determining) the gender of a subject from an image captured by a camera 11, each image captured by the camera 11 of multiple subjects is input to the input layer of the simulated network, and a corresponding answer (label) is presented, and the weight values ​​are updated and learned at the time of feedback. By performing such learning for a large number of subjects, the feature amount of each subject is reflected in the parameters, improving the accuracy of the determination.

[0027] In addition, it is preferable to simultaneously train the signals read from the modulation element and the imaging element 4 and the recognition unit 12 using, for example, a deep learning framework. In this case, it is preferable to train the deep learning using an adversarial learning framework so that the captured image has as little visual meaning as possible, thereby making it possible to capture images that are incomprehensible to humans and even incapable of personal recognition by the recognition unit 12 without compromising the recognition function. In this way, a set of the camera 11 and the recognition unit 12 is designed by optimizing the parameters of the recognition unit 12, which is software, and the hardware design, which is the pattern of the modulation element 2, in association with each other within the machine learning framework.

[0028] In this way, by essentially designing a modulation pattern in such a way that a light ray that has passed through one light-transmitting section is incident on multiple pixels, or in such a way that each light ray that has passed through multiple light-transmitting sections is incident on one pixel, it is possible to produce a modulation element that makes it impossible to recognize an individual, while enabling the recognition of ancillary information about the individual.

[0029] The present invention also includes the following aspects.

[0030] (1) The present camera 11 can also be constructed by disposing the present modulation element on either the front or rear surface of the photographic lens of a normal camera. In this case, the modulation element should be designed to modulate the optical image taking into account the imaging performance of the photographic lens.

[0031] (2) Specific intended uses of the imaging system 1 include gender determination, age determination, gestures (actions), personal ID, and various other types of additional information that do not lead to the identification of the subject. The determination result can be notified by further providing a display, speaker, etc. that displays the determination result from the recognition unit 12. The imaging system 1 can also be applied to individuals such as animals other than humans. Therefore, the imaging system 1 can be applied not only as a portable system, but also as a stationary system.

[0032] (3) The modulation pattern on the surface of the modulation element may be irregular, or one or more types of modulation patterns may be arranged repeatedly in at least one of the vertical and horizontal directions for each size. The size of the split modulation pattern depends on the recognition application, but may be a size corresponding to an area of ​​several tens to several hundreds of pixels 42, for example, an array area of ​​100 x 100 pixels, or less or more than that, in relation to the number of pixels 42. In addition, it may include an embodiment in which pinholes as shown in Figure 3(B) are formed adjacent to each other as part of the pattern of the modulation element, and light rays passing through both pinholes are guided to the same pixel.

[0033] (4) The mask 2 can be made of a material that changes the modulation pattern, such as a liquid crystal display (LCD) panel, instead of a fixed type. By making the modulation pattern variable, it can be switched and controlled by an electrical signal to become a preset pattern depending on the application, and can also be switched over time for the same application, and in either case, the privacy performance can be improved.

[0034] We then present experimental data on how we (A) model lensless acquisition and evaluate various lensless imaging methods, (B) demonstrate visual privacy preservation through custom loss functions for human and machine vision and a method for training unique pairs of coded patterns and recognition features, (C) present our experiments along with hardware realizations, and (D) discuss our experimental conclusions.

[0035] (A) Safe lensless imaging We first provide a background on lensless imaging for visual privacy protection and imaging systems for face recognition.

[0036] (1) Coded Lensless Image Lensless imaging is a new technique to capture images without using complex lens systems. A coded pattern is used to modulate the incident light at single or multiple pixels. The latter approach is more common as it allows for the capture of a single shot image without modifying the pattern. Lensless imaging is illustrated in Figure 6 for short (A) and long (B) distances d1 between the image and coded planes. Given a scene x and a coded pattern H, the lensless measurement y is expressed as (Equation 1):

[0037]

number

[0038] where * is the convolution operator and η is additive noise. When the distance d1 is small, the camera can be thin like FlatCam (i.e., a camera that can shoot without a lens), but the angle of the incident light ray is also limited by the field of view of the pixels of the sensor 4. When the distance is large, the field of view is defined by the entrance pupil of the camera, the diameter of the mask 2. For the same resolution as the binary pattern H and a large kernel size, increasing the distance d1 blurs the image and improves visual privacy protection. Therefore, a large distance d1 is adopted. We learn the binary pattern H by modeling the coded imaging as a binary convolution.

[0039] (2) Lensless imaging system for face recognition The imaging system 1 shown in FIG. 7 uses a lensless camera 11 equipped with a mask 2 and a sensor 4 to capture images and transmit them without reconstruction to a recognition unit 12 based on ResNet18 (a convolutional neural network with a depth of 18 layers).

[0040] First, we evaluated imaging scenarios including conventional coded imaging (with fixed and trained patterns). For fixed lensless imaging, we used a pinhole, a defocused pattern, and a random pattern without reconstruction (Rand-woRec). For trained lensless imaging, patterns were trained without constraints and without training (LwoC-woRec). The reconstruction network is described later.

[0041] [Table 1]

[0042] Table 1 shows the Top1 accuracy (%) of various sampling methods using ResNet18. For LwC-MSE, α = 10 -8 ,For LwC-TV, α=10 -6 Note that Top1 accuracy (%) is an expression of the recognition rate, which refers to the recognition rate of the first candidate. As shown in the results in Table 1, either conventional imaging or pinhole imaging achieves the best accuracy. Defocus and randomly coded imaging cause a loss of accuracy of 20% to 40%.

[0043] As shown in Figure 7, it is best for the recognition result b to be correct, but at the same time, it is required that the captured image y is blurred (unintelligible to humans). Simply optimizing to improve the recognition rate means that the captured image y will not be blurred (pinehole has good performance in Table 1), but blurring y means that the recognition rate will decrease (Defocus and random have poor performance in Table 1). This method solves this tradeoff by simultaneously optimizing the mask 2 pattern (for blurring) and the recognition unit 12. LwC-TV has high performance even though the image is blurred, and in some cases has better performance than pinehole. In other words, it realizes pattern generation that is intelligible to machines even if not understood by humans.

[0044] Also, Figure 8 shows a visual comparison of various imaging systems, where the scale above the measurements and patterns indicates the ratio of blur to exposure. In Figure 8, traditional pinhole imaging reveals image details that imaging with defocus and random patterns does not. Thus, there is a trade-off between accuracy and visual privacy protection. That is, although the learning pattern significantly improves recognition accuracy with a loss of about 5% compared to pinhole and traditional imaging, it does not guarantee visually protected measurements. As shown in Figure 8, when the coded ratio r is small (i.e., r = 1 / 16), LwoC-woRec reveals the subject's identity. Therefore, a method to control the trade-off between accuracy and privacy is desired.

[0045] (B) Safe learning lensless imaging (1) Protecting privacy from human vision To make it impossible to identify people from lensless images, we wanted to learn an encoding pattern so that the captured image would be the same as the image captured with defocused patterns while maintaining high recognition capabilities. To achieve this, we maximize the blur of the captured image by minimizing the mean squared error (MSE) in (Equation 2).

[0046]

number

[0047] Here, l m denotes a matrix of all 1s. This is the coded pattern for defocused imaging. Conversely, as shown in Figure 8, the training pattern may converge to a smaller local region (or smaller variation). Thus, the measurements convolve from smaller regions of the image and reveal more information. As a result, we maximize the total variation (TV) of the coded pattern as in (Equation 3).

[0048]

number

[0049] Here, Δ x and Δ y denote the horizontal and vertical gradient operators, respectively. When using the TV loss, the training patterns need to be more diverse than when using the MSE loss.

[0050] (2) Protecting privacy from machine vision In security applications, the pair of pattern Hi and recognition function Ri must be unique. In other words, a correct {Ri, Hi} has a high recognition function, but a mismatched pair {Ri, Hj} has a low recognition function. To give a more understandable example, this shows that the pattern Hi and recognition function Ri act like a key. In other words, the accuracy is high only when the key Hi and the keyhole Ri match, and the accuracy is low when they do not match. Even if a certain key Hi and keyhole Ri are public keys, if i is unknown, even if an image taken with Hi is eavesdropped, the Ri of that combination cannot be identified, so information cannot be directly eavesdropped from there. By applying this, for example, if Hi is changed over time on an LCD panel and Ri on the server side is synchronized with this, it can be applied like a code table for an ATM (Automatic Teller Machine), further enhancing security.

[0051] When optimized as in (B).(1) above, a pattern that is not obvious to humans but is easy for machines to understand is generated. In other words, there is a possibility that the image will be easily detectable by any learning device (for example, as an extreme example, a mask with horizontal stripes for person A and vertical stripes for person B). To prevent this, the condition in (Equation 4) below is added, and an image encoded with a certain pattern Hi is optimized so that it can be discriminated only by Ri optimized at the same time, and is difficult to discern with other Ri, so that it cannot be discriminated by a recognition function Rj that does not know the pattern Hi. In other words, a pattern of mask 2 that simultaneously achieves recognition rate, blurring, and machine privacy (the correlation between changes in the captured image and the label becomes difficult to understand) is generated.

[0052] For example, a plurality of types of patterns Hi and recognition functions Ri optimized for various patterns are stored (prepared) in advance as combinations in a storage unit (not shown), for example, a storage unit in the recognition unit 12, and a control unit (not shown) stores and controls this combination information. When the recognition unit 12 or the control unit (not shown) selects the mask 2 of the pattern Hi during a certain photographing, the recognition function Ri as a set is selected and applied to the recognition process, rather than the uncorresponding recognition function Rj, thereby executing the recognition process in the intended, i.e., optimized state. In this way, by being applicable like a code table, security can be further improved.

[0053] However, the aforementioned method only protects privacy from human vision, and training multiple examples will generate similar pattern-recognition feature pairs. This can be seen in Figure 9, which shows high accuracy for the diagonal. Note that Figure 9 shows the confusion matrix of LwoC-woRec for multiple Hi and Ri, where each value of (i,j) indicates the Top1 accuracy of the j-th recognition feature Rj on the input lensless measurement of the i-th coding pattern Hi. Privacy protection of machine vision is required to train unique pairs {Ri,Hi}. regIf represents the cross-entropy loss function of the input x and the label b, it is easy to reduce the accuracy of mismatched pairs by (Equation 4).

[0054]

number

[0055] (Equation 4) requires extensive computation with multiple inferences of Ri as the number of unique pairs M grows. Finally, the training loss is a combination of the human vision and machine vision visual privacy-preserving losses, as expressed in (Equation 5).

[0056]

number

[0057] For a new pair of coded patterns H and R, a more complex loss is added.

[0058] (C) Experimental results of simulation data (1) Dataset and training (1-1) Dataset Here we present the main results on the VGG-Face2 dataset (pre-trained model). We also performed additional experiments on the calibrated Microsoft® Celeb (MS-Celeb) and CASIA datasets. For all datasets, we selected the 10 classes with the highest number of images and split them into training and test sets in a 95:5 ratio. Random cropping and vertical flipping were employed to supplement the data.

[0059] (1-2) Training Here, we used ResNet18 for face recognition. The network was trained using a stochastic gradient descent optimizer. The mini-batch size was 128. Three settings were used: image size n = {63 × 63, 127 × 127}, and coded pattern size m = {32 × 32, 64 × 64}. The coded ratio is defined as r = n / m, and the aperture ratio is expressed as the total number of "1" elements in the pattern relative to the entire pattern area. After training, the network with the highest Top1 test accuracy was selected as the final solution. For the weight coefficients α and β, we used a 10 -2 From 10 -8 We tested various combinations up to 17 residual blocks. For reconstruction, we learned the residuals between clean and captured images from the Div2K (training and test images) dataset using 17 residual blocks.

[0060] (2) Human visual privacy performance Visual privacy is very difficult to evaluate due to the lack of research on how to measure the ability of the human eye to recognize objects. In general, if an image is blurry, humans have a hard time recognizing the subject. Therefore, we adopted a non-reference blur metric to evaluate the visual privacy quality. As shown in Table 1 above, all learning pattern schemes produced high recognition accuracy with less than 5% loss compared to traditional pinhole imaging. Furthermore, reconstruction is not necessary for recognition, but it reduces accuracy. It should be noted that better reconstruction methods would increase accuracy. However, they require fixed coding patterns, which is not suitable for our method. Conversely, reconstruction midway through the process may increase security risks. Furthermore, it has been recently suggested that direct recognition is also better than initial reconstruction.

[0061] From Figure 10(A) and (B), it is easy to observe that the MSE loss provides a trade-off between defocused imaging and unconstrained imaging (LwoC-woRec), while the TV loss has a trade-off between Rand-woRec and LwoC-woRec. The smaller the weight, the closer the result is to the unconstrained result. As the curve moves to the upper right, the TV loss gives slightly better results than the MSE loss. Note that in Figure 10(A) and (B), the mask patterns are the same 32x32 for both, but the image sizes are different (the amount of information varies depending on the number of pixels), resulting in different recognition rates. (B) has a higher resolution than (A), so the recognition rate is higher even with the same amount of optical blur.

[0062] The effect of the weighting coefficient is shown in Figure 11. The smaller the weight, the smaller the aperture ratio and the higher the accuracy, but more information is revealed. Visually, both MSE and TV loss functions can ensure visual privacy at the expense of accuracy. Conversely, decreasing the aperture ratio reduces the optical efficiency. Although we do not consider this effect in our simulations, it significantly affects the recognition accuracy from real measurements.

[0063] The results of this experiment show that the weighting coefficient α is 10 -4 ~10 -6 And the MSE loss is 10 -6 ~10 -8 Based on our experiments, we recommend α=10 for a good tradeoff between performance and privacy. -4 We choose the TV loss of α=10 for higher accuracy. -5 was selected.

[0064] (3) Machine visual privacy and security performance For security applications, we define two objective scores for the confusion matrix of patterns and recognition features. We call these self-accuracy and mutual accuracy. Self-accuracy is shown in (Equation 6) and is defined as the average of the diagonal of the confusion matrix. It is the average accuracy using the correct pair H and R.

[0065]

number

[0066] Cross-accuracy is the average accuracy of the off-diagonal lines of the confusion matrix, and represents the performance when mismatched pairs of training patterns and recognition features are used. In general, a high self-accuracy and a low cross-accuracy are desirable. The larger the difference between the performance of self-accuracy and cross-accuracy, the better. The confusion matrices of various methods are shown in Figure 12.

[0067] Table 2 also shows the Top1 accuracy (%) of various sampling methods using ResNet18, with α = 10 for LwC-MSE. -8 ,For LwC-TV, α=10 -6 ,For LwC-TV-Reg, α=10 -4 , β=10 -6 It is.

[0068] [Table 2]

[0069] As the results in Table 2 show, in the absence of constraints, LwoC-woRec achieves the highest self-accuracy, but also has a higher mutual accuracy. The loss of human vision due to MSE and TV improves visual privacy for human vision, but does not help protect against machine vision. Therefore, high values ​​of average (70%) and maximum (80%) mutual accuracy are reported. Conversely, LwoC-woRec, as shown in (Eq. 4), reg mvThe loss of ML loss for visual privacy preservation in machine vision helps to reduce the mutual accuracy while maintaining high accuracy. The ML loss is effective up to M=3, with a 40% accuracy gap between self-accuracy and mutual accuracy, compared to 18% for LwoC, 4% for LwC, and 12% for LwC-TV. Unfortunately, as the number of unique pairs in M ​​increases, the effectiveness of the ML loss decreases as the mutual accuracy increases. One reason is that the training framework is sequential, making it more difficult to train new unique pairs. However, the accuracy is also heavily affected by the hyperparameters α and β, which have not yet been optimized.

[0070] (4) Experimental results using real data (Hardware Realization) To verify the proposed method, we implemented a prototype imaging system as shown in Fig. 13. This camera consists of a monochrome imaging sensor 4 (Grasshoper3 model GS3-U3-41C6M-C, 2048 × 2048) and a mask 2B. The mask 2B consists of a spatial light modulator 20B (SLM; LC 2012, 1024 × 768) and polarizers 20f and 20b arranged before and after the spatial light modulator 20B, and the incident light is intensity modulated by adjusting the relative angle between the two. The distance between the sensor 4 and the code surface of the mask 2B is about 17 mm. A monitor (Plasmadisplay) that displays the image is installed about 1 m away from the SLM.

[0071] The coded patterns were rescaled from 32 × 32 to 716 × 716 and padded with zeros so that the size of the SLM was 1024 × 768. Five different coding patterns were evaluated for mask 2, as shown in Figure 14. The shutter time was manually selected to compensate for the difference in aperture ratio. The face test image was also rescaled and calibrated on the display screen to appear in the center of the image sensor. However, there is still interreflection between the image sensor and the SLM. Therefore, a captured image with the aperture of the SLM close was used for the correction. In addition, to reduce the effects of noise and poor light efficiency, an average of 10 times the captured measurements were adopted as the input for the recognition function.

[0072] First, measurements were captured in 16-bit grayscale for various patterns shown in Figure 14. Unlike the simulation, in a real imaging scenario, pinhole imaging is of very low quality due to very little light. A visible image can also be observed in the capture. Similar to the simulation, no privacy information was observed from the measurements with defocus and random patterns (50% exposure rate). Furthermore, without constraints, the learning pattern LwoC revealed more information than the TV loss constraint.

[0073] For face recognition applications, we selected a subset of 70 and 20 highest resolution images in the CASIA train and test sets, respectively, to capture real lens-free measurements. Prior to face recognition, the real captured images were normalized and further cropped to 80% of the central face region. A background image was captured for each image using an all-zero mask. The background image was subtracted to correct for light leakage. The final training images were resized to 128x128 for training. We further refined the model to fit real images (real images) by retraining the simulated resNet18 with the above real-world shooting data.

[0074] Although it performs well in simulations, we can see that pinhole imaging performs poorly on real datasets due to inefficient light capture at low coded ratios. Pinhole images are noisier than other images, limiting performance. Pinhole images also contain many details with a small blur score of 0.140. Defocused imaging leads to poorer recognition performance. It is shown that the captured images have a small blur score due to the lack of information. Random masks also performed a little better, but are still worst than the trained masks of LwoC and LwC-TV.

[0075] [Table 3]

[0076] Table 3 shows the Top1 accuracy (%) using the selected CASIA10 surface dataset. Table 3 shows the experimental results for real images, which show that the performance of the proposed LowC-TV is sufficiently high even when the image is blurred, i.e., the apparent privacy is protected, as in the simulation. It was also shown that the decrease in image contrast in the real implementation can be improved by using background subtraction (subtracting the brightness value of an image with nothing in it from the captured image).

[0077] (D) Conclusion and Discussion In this paper, we proposed a learned lensless imaging system to protect visual privacy from both human vision models and the target machine vision model. To protect visual privacy from human vision, we used MSE to maximize the blurring of measurements and TV loss to maximize the variation of the learned patterns. Through our experiments, we confirmed that our method enables the trade-off between visual privacy protection and recognition accuracy in lensless imaging. Although the accuracy is slightly reduced, our method can sufficiently protect visual privacy. Furthermore, we used recognition loss to protect visual privacy from the machine vision model. A sequential training framework was presented to enable security applications by training multiple unique pairs of coded patterns and deep learning-based recognition functions.

[0078] Here, we are based on a simple hypothesis that the less blurry the image, the less likely it is for humans to recognize the object. However, the threshold blur metric for object recognition is not clear and depends on the coded ratio. Meanwhile, blind image deblurring techniques can be used to reconstruct the original image. Further research on subjective quality assessment and the impact of learned kernels is encouraged.

[0079] Our incremental training method could learn unique pairs of coded patterns and recognition features. However, the framework is limited in the number of unique pairs (i.e., key space) that it can handle. It remains an open problem how to handle the case of a large number of unique pairs of H and R (i.e., increasing M). Furthermore, techniques on adversarial samples can be further integrated to provide a better training method. Unlike previous techniques that used fixed patterns, we learn coded patterns to achieve higher recognition accuracy. However, the system was trained only on simulated data.

[0080] As described above, the camera of the present invention includes a planar imaging element in which a plurality of pixels made of photosensitive elements are arranged, and a planar modulation element arranged in front of the imaging element and having a pattern formed thereon for modulating incident light, the pattern being characterized in that it includes an array of a plurality of light-transmitting sections each of which guides light rays from a subject coming from a plurality of directions to one pixel.

[0081] According to the present invention, the light from the subject is optically modulated by the modulation element, and then captured by the imaging element. The captured image has spatial projection information such as an optical retinal image destroyed, but the information essentially necessary for recognition can be retained. This makes it difficult to visually recognize the contents from a recorded or leaked data string, so privacy is protected.

[0082] In addition, it is preferable that the light transmitting portion includes a portion that guides light rays from a plurality of directions to a plurality of pixels. According to this configuration, by dispersing light rays transmitted through one light transmitting portion, spatial projection information is further destroyed, and privacy is protected.

[0083] It is also preferable that the light transmitting portion is a mask surface for blocking light and has holes drilled therein. With this configuration, the modulation element can be easily produced by drilling the holes.

[0084] Moreover, it is preferable that the hole is larger than the size of the pixel. With this configuration, a plurality of light rays can be transmitted, and the spatial projection information is destroyed accordingly.

[0085] In addition, it is preferable that the light transmitting portion is a light transmitting plate-like body having an uneven surface. With this configuration, it is possible to produce a modulation element by using a method other than a mask, for example, by surface processing of a light transmitting member.

[0086] In addition, the imaging system according to the present invention preferably includes a readout unit that reads out the image of the subject captured by the camera, and a recognizer that performs predetermined recognition of the auxiliary information of the subject from the readout image. According to the present invention, the image is directly recognized by the recognizer without restoring it to a retinal image, which is advantageous in terms of privacy protection since it does not involve any visually understandable image at all.

[0087] In addition, it is preferable that the modulation element and the recognition unit are optimized in terms of both the degree of blurring of the image of the subject captured through the pattern of the modulation element and the recognition rate of the recognition unit. With this configuration, it is possible to simultaneously process the blurring of the captured image through the pattern and the recognition rate in the recognition unit in the best condition.

[0088] It is also preferable to have a storage unit that stores combinations of multiple types of patterns Hi (i=1, 2, ...) and recognition functions Ri optimized for each pattern, and a control unit that selects the combination of patterns and recognition functions (Hi, Ri) when capturing an image. This configuration can be applied like a so-called code table, further enhancing security. [Explanation of symbols]

[0089] 1 Imaging system 11 Camera 12 Recognition unit (recognizer) 2, 2a, 2b, 2c, 2B Mask (modulation element) 20B Spatial light modulator (modulation element) 21,21c, 21b, 22b, 21c Transparent part 2d Plate (modulation element) 21d Rough surface (partially transparent) 4. Image sensor 42 pixels

Claims

1. An imaging system including a camera, a readout unit, and a recognition unit, The camera includes: a planar imaging element in which a plurality of pixels each made of a photosensitive element is arranged; a planar modulation element disposed in front of the imaging element and having a pattern formed thereon for modulating incident light; the pattern includes an array of light transmitting portions each of which guides light rays from a plurality of directions, among light rays from a subject, to one pixel; The reading unit reads out an image of a subject captured by the camera, The recognition unit performs a predetermined determination on auxiliary information of the subject from the read captured image, An imaging system characterized in that the modulation element and the recognizer are optimized in terms of both the degree of blur of the image of the subject captured through the pattern of the modulation element and the recognition rate of the recognizer.

2. A storage unit that stores combinations of a plurality of types of patterns Hi (i=1, 2, . . . ) and recognition functions Ri optimized for each type of pattern; 2. The imaging system according to claim 1, further comprising a control unit which selects a combination pattern and a recognition function (Hi, Ri) during imaging.

3. The imaging system according to claim 1 or 2, wherein the pattern includes a pattern that guides light rays transmitted through a common light-transmitting portion to a plurality of pixels.

4. 4. The imaging system according to claim 1, wherein the light transmitting portion is a mask surface for blocking light and holes are formed in the mask surface.

5. 4. The imaging system according to claim 1, wherein the light-transmitting portion is a light-transmitting plate-like body having an uneven surface.

Images