Lensless camera and control method therefor

Abstract

The present disclosure relates to making it practically impossible to restore image data of a lensless camera through artificial intelligence even if a plurality of output images and original images output from the lensless camera are acquired, and can provide a lensless camera comprising: a variable light modulation mask; an image sensor; a memory for storing data related to a plurality of point spread functions (PSFs); and a processor, which selects one from among the plurality of PSFs so as control that the variable light modulation mask has a light modulation pattern of the selected PSF, generates an output image by performing image processing on an input image light-modulated according to the light modulation pattern of the selected PSF, and controls that the output image includes PSF identification information corresponding to the selected PSF.

Description

Lensless camera and its control method

[0001] The present disclosure relates to a lensless camera, and more particularly, to a lensless camera advantageous for image encryption and a method for controlling the same.

[0002] A typical camera is a lens camera, with a lens positioned in the direction in which light enters the camera, so that point light sources in each scene are focused as points on the image sensor through the lens to acquire an image.

[0003] Recently, research on lensless cameras without lenses has been actively conducted.

[0004] Lensless cameras replace lenses with light-modulating masks (or pattern masks) positioned in the direction of light incidence. The incoming point light is modulated by the mask and imaged onto the image sensor. This method of acquiring images without the use of a lens is called lensless imaging.

[0005] There are various types of optical modulation masks for lensless cameras, and diffuser-type masks are common. Representative examples include phase masks that use phase modulation and amplitude masks that use amplitude modulation.

[0006] Phase masks are more efficient in terms of light collection than amplitude masks. Furthermore, they feature shift-invariant imaging, which allows for imaging based on a unique pattern. Therefore, phase-mask-based lensless imaging offers the advantage of providing a more robust imaging model than amplitude-mask-based lensless imaging.

[0007] Instead of focusing a lens, a lensless camera installs a light-modulating mask that forms a specific pattern in front of the image sensor, allowing light to pass through and create an image. The light-modulating mask modulates the path of the passing light according to the Point Spread Function (PSF), which exhibits a unique pattern determined by its shape structure. This allows the image transmitted to the image sensor to be focused in a blurred manner, obscuring the original image information. Based on this fundamental principle, lensless cameras can be applied as a security technology that can conceal personal information in the output image.

[0008] However, if an artificial intelligence model, such as a deep learning model that acquires a large number of output images and original images from a lensless camera and restores the images, is trained, the trained artificial intelligence model can perform the function of inversely transforming the PSF of the lensless camera, thereby converting the output image into the original image.

[0009] In other words, anyone can use artificial intelligence to restore the image data of a lensless camera used for security purposes as long as they can obtain a large number of output images and original images from the lensless camera, which can be fatal to the security purpose of the lensless camera.

[0010] The present disclosure is proposed to solve the aforementioned problem, and its purpose is to provide a lensless camera and a control method thereof that make it virtually impossible to restore image data of a lensless camera through artificial intelligence even when a large number of output images and original images are acquired from the lensless camera.

[0011] In order to achieve the above object, the present disclosure may provide a lensless camera including a variable optical modulation mask, an image sensor, and a memory storing data regarding a plurality of PSFs (Point Spread Functions), and a processor controlling the variable optical modulation mask to have a optical modulation pattern of the selected PSF by selecting one of the plurality of PSFs, and performing image processing on an input image optically modulated according to the optical modulation pattern of the selected PSF to generate an output image, wherein the output image includes PSF identification information corresponding to the selected PSF.

[0012] The processor can control the optical modulation pattern of the variable optical modulation mask to be periodically changed to another optical modulation pattern among the plurality of PSFs.

[0013] The above variable optical modulation mask may include a transmissive display element.

[0014] The above PSF identification information can be encrypted and included in the output image using steganography.

[0015] In addition, to achieve the above purpose, the present disclosure may provide an electronic device including an image input unit, a memory storing a plurality of restoration models corresponding to each of a plurality of PSFs, and a control unit that receives a photographed image captured by a lensless camera through the image input unit and controls restoration of the photographed image using a restoration model corresponding to PSF identification information extracted from the photographed image among the plurality of restoration models.

[0016] Each of the above plurality of restoration models can be generated by artificial intelligence learning of input / output image learning data prepared in advance according to each of the above plurality of PSFs.

[0017] In addition, to achieve the above object, the present disclosure may provide a method for controlling a lensless camera, including the steps of selecting one of a plurality of PSFs and controlling a variable optical modulation mask to have an optical modulation pattern of the selected PSF, performing image processing on an input image optically modulated according to the optical modulation pattern of the selected PSF to generate an output image, wherein the output image includes PSF identification information corresponding to the selected PSF, and restoring the output image using a restoration model corresponding to the PSF identification information extracted from the output image among a plurality of restoration models.

[0018] The effects of the lensless camera and its control method according to the present disclosure are as follows.

[0019] According to one aspect of the present disclosure, by using a variable optical modulation mask capable of implementing a plurality of point spread functions, there is an advantage in that it is virtually impossible to restore image data of a lensless camera through artificial intelligence even when a plurality of output images and original images are acquired from a lensless camera.

[0020] FIG. 1 is a block diagram of a lensless camera according to one aspect of the present disclosure.

[0021] Fig. 2 is a block diagram of an electronic device for restoring the output image of the lensless camera of Fig. 1.

[0022] FIG. 3 is a flowchart illustrating the operation of a lensless camera according to one aspect of the present disclosure.

[0023] FIG. 4 is a flowchart regarding the operation of an electronic device according to one aspect of the present disclosure.

[0024] FIG. 5 is an example of an input image, an output image, and a restored image according to one aspect of the present disclosure.

[0025] FIG. 6 is another example of an input image, an output image, and a restored image according to one aspect of the present disclosure.

[0026] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0027] These components may each be implemented as separate individual hardware modules, or may be implemented as two or more hardware modules, or two or more components may be implemented as one hardware module, and in some cases, they may also be implemented as software.

[0028] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0029] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0030] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprises" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the disclosure, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] In this disclosure, the expression “at least one of A and B” may mean “A,” may mean “B,” or may mean both “A” and “B.”

[0032] Referring to FIG. 1, a lensless camera according to one aspect of the present disclosure will be described. FIG. 1 is a block diagram of a lensless camera according to one aspect of the present disclosure.

[0033] The above lensless camera (100) may include a variable light modulation mask (110), an image sensor (120), a memory (170), and a processor (180).

[0034] It goes without saying that the lensless camera (100) described in the present disclosure may include other components in addition to the components listed above.

[0035] The variable light modulation mask (110) may be configured as a transmissive display element (or transmissive image element) capable of changing the shape of a light modulation pattern. Examples of the transmissive display element may include a Liquid Crystal Display (LCD), a Spatial Light Modulator (SLM), etc. Data regarding a plurality of light modulation patterns for the variable light modulation mask (110) may be stored in the memory (170). The light modulation pattern of the variable light modulation mask (110) may be changed to any one of the plurality of light modulation patterns under the control of the processor (180). When one light modulation pattern is displayed on the transmissive display element and the displayed light modulation pattern is changed to another light modulation pattern, the transmissive display element may function as the variable light modulation mask.

[0036] The above variable optical modulation mask (110) can optically modulate the input image (P1) according to one of the plurality of optical modulation patterns and output it to the image sensor (120).

[0037] The image sensor (120) can sense the photo-modulated image and provide it to the processor (180). The image sensor (120) can include, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, etc.

[0038] The above memory (170) stores data supporting various functions of the lensless camera (100). The memory (170) can store a plurality of application programs (or applications) running on the lensless camera (100), data for the operation of the lensless camera (100), and commands. In particular, the memory (170) can store data regarding a plurality of light modulation patterns (i.e., a plurality of PSFs) for the variable light modulation mask (110). Each of the plurality of light modulation patterns can have a different PSF with low similarity to each other.

[0039] The processor (180) can control the overall operation of the lensless camera (100). The processor (180) can image process the sensed image to generate an output image (P2). In particular, the processor (180) can control the optical modulation pattern of the variable optical modulation mask (110) to change from time to time. For example, the processor (180) can sequentially or randomly select one of the plurality of optical modulation patterns at predetermined intervals and control the variable optical modulation mask (110) to have the selected optical modulation pattern.

[0040] It can be seen that the above output image (P2) is an encrypted image that cannot be identified due to the characteristics of a lensless camera.

[0041] The restoration of the above encrypted output image (P2) will be described with reference to Fig. 2. Fig. 2 is a block diagram of an electronic device for restoring the output image of the lensless camera of Fig. 1.

[0042] The electronic device (200) may include a video input unit (120), a display unit (150), a memory (170), and a control unit (180). For example, the electronic device (200) may include a mobile phone, a smart phone, a desktop computer, a laptop computer, a tablet PC, an ultrabook, a wearable device, etc.

[0043] The above image input unit (120) can receive the output image (P2) of the lensless camera (100). The above image input unit (120) can receive the output image (P2) by wire or wirelessly.

[0044] When the above output image (P2) is input via a wire, the image input unit (120) may include a wired interface unit such as an image I / O (Input / Output) port.

[0045] When the above output image (P2) is input via a wire, the image input unit (120) may include a wireless communication unit for wireless communication with the lensless camera (100).

[0046] The display unit (250) displays (outputs) information processed in the electronic device (100). For example, the display unit (151) may display execution screen information of an application program running in the electronic device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information. In particular, the display unit (250) may display the output image (P2) and the restored image (P3) restored by the electronic device (200).

[0047] The above memory (270) can store a program for the operation of the control unit (280), and can also store input / output data (e.g., still images, moving images, etc.). In particular, the memory (270) can store a restoration model (275) for each of the plurality of optical modulation patterns. Each restoration model can be generated by learning a large amount of input images and output images corresponding to each optical modulation pattern through an artificial intelligence algorithm.

[0048] The above control unit (280) can control the overall operation of the electronic device (200). The above control unit (180) can perform a restoration operation on the output image (P2) using a restoration model corresponding to the output image (P2) to generate a restored image (P3).

[0049] Although the image quality of the above restored image (P3) may be somewhat deteriorated in various aspects such as detail and color compared to the above input image (P1), information for image recognition such as person detection and situation recognition can be extracted.

[0050] Let's take a closer look at the artificial intelligence mentioned above.

[0051] Artificial intelligence (AI) is the study of artificial intelligence or the methodologies for creating it, while machine learning (ML) defines various problems in the field of AI and studies the methodologies for solving them. Machine learning is also defined as an algorithm that improves performance on a task through consistent experience.

[0052] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities, comprised of artificial neurons (nodes) formed by the connection of synapses. An ANN can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

[0053] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0054] Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0055] The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

[0056] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0057] Supervised learning refers to a method for training an artificial neural network when given labels for the training data. The labels can refer to the correct answer (or output value) that the artificial neural network must infer when the training data is input to the artificial neural network. Unsupervised learning can refer to a method for training an artificial neural network when the training data is not given labels. Reinforcement learning can refer to a learning method in which an agent defined within a given environment is trained to select actions or action sequences that maximize the cumulative reward in each state.

[0058] Machine learning implemented with a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a subset of machine learning. Hereinafter, the term "machine learning" is used to encompass deep learning.

[0059] Object detection models using machine learning include the single-stage YOLO (You Only Look Once) model and the two-stage Faster R-CNN (Regions with Convolution Neural Networks) model.

[0060] The YOLO (You Only Look Once) model is a model that can predict objects and their locations within an image by looking at the image only once.

[0061] The YOLO (You Only Look Once) model divides the original image into grids of equal size. For each grid, it predicts the number of bounding boxes in a predefined shape centered around the grid center, and calculates a confidence level based on this prediction.

[0062] Afterwards, whether the image contains an object or is just a background is included, and locations with high object confidence are selected so that the object category can be identified.

[0063] The Faster R-CNN (Regions with Convolution Neural Networks) model is a model that can detect objects faster than the RCNN model and the Fast RCNN model.

[0064] This article specifically explains the Faster R-CNN (Regions with Convolution Neural Networks) model.

[0065] First, feature maps are extracted from the image using a Convolution Neural Network (CNN) model. Based on the extracted feature maps, multiple regions of interest (RoIs) are extracted. RoI pooling is performed for each region of interest.

[0066] RoI pooling is a process of setting a grid to a predetermined size of H x W for the feature map onto which the region of interest is projected, extracting the largest value for each cell included in each grid, and extracting a feature map with a size of H x W.

[0067] A feature vector is extracted from a feature map having a size of H x W, and object identification information can be obtained from the feature vector.

[0068] In FIGS. 1 and 2, the lensless camera (100) and the electronic device (100) are depicted as separate devices. However, the lensless camera (100) and the electronic device (100) may be configured as a single device (e.g., a camera, a smartphone, a tablet PC, a laptop, etc.). The lensless camera (100) may also perform the operations of the electronic device (100).

[0069] Hereinafter, the operation of the lensless camera (100) will be described with reference to FIG. 3. FIG. 3 is a flowchart regarding the operation of the lensless camera according to one aspect of the present disclosure.

[0070] First, data regarding a plurality of PSFs may be pre-stored in the memory (170) of the lensless camera (100) [S31]. The plurality of PSFs may be prepared so that they have low similarity to each other, making it impossible to infer another PSF from one PSF.

[0071] The processor (180) of the lensless camera (100) can sequentially or randomly select one of the plurality of PSFs at predetermined intervals and control the variable light modulation mask (110) to have a light modulation pattern according to the selected PSF [S32].

[0072] The above variable optical modulation mask (110) optically modulates the input image (P1) according to the optical modulation pattern, and the image sensor (120) can capture the optically modulated input image [S33].

[0073] The above processor (180) can generate an output image (P2) by performing image processing on the photo-modulated input image to include information for identifying the PSF at the time when the photo-modulated input image was captured (i.e., the PSF used to photo-modulate the input image), i.e., PSF identification information [S34]. For example, the PSF identification information can be included in the output image (P2) using a steganographic method.

[0074] Hereinafter, the operation of the electronic device (200) will be described with reference to FIG. 4. FIG. 4 is a flowchart regarding the operation of the electronic device according to one aspect of the present disclosure.

[0075] The control unit (280) of the electronic device (200) can generate a restoration model for each PSF by performing artificial intelligence learning on input / output image learning data prepared in advance according to each of the plurality of PSFs and store the model in the memory (270) [S41]. The artificial intelligence learning of the input / output image learning data and the generation of the restoration model for each PSF may be performed in an external device (or external server). In this case, the control unit (280) can receive the restoration model for each PSF from the external device and store the model in the memory (270).

[0076] The above control unit (280) can receive the output image (P2) through the image input unit (220) [S42].

[0077] The above control unit (280) can extract the PSF identification information from the output image (P2) [S43].

[0078] The above control unit (280) can restore the output image (P2) to the restored image (P3) using a restoration model corresponding to the PSF identification information.

[0079] Referring to Fig. 5, we will examine a case where the output image (P2) is restored using a restoration model corresponding to the PSF used when the input image (P1) is photomodulated. Fig. 5 is an example of an input image, an output image, and a restored image according to one aspect of the present disclosure.

[0080] As shown in Fig. 5, it can be seen that the output image (P2) is an encrypted image that cannot be identified when compared to the input image (P1).

[0081] When the output image (P2) is restored using a restoration model corresponding to the PSF used when the input image (P1) is photo-modulated, the restored image (P3) has somewhat deteriorated image quality in various aspects such as detail and color compared to the input image (P1), but it can be seen that information for image recognition such as person detection and situation recognition can be extracted.

[0082] With reference to Fig. 6, we will examine a case where the output image (P2) is restored using a restoration model other than the restoration model corresponding to the PSF used when the input image (P1) is photomodulated. Fig. 6 is another example of an input image, an output image, and a restored image according to one aspect of the present disclosure.

[0083] As shown in Fig. 6, when the output image (P2) is restored using a restoration model other than the restoration model corresponding to the PSF used when the input image (P1) is photomodulated, it can be seen that the restored image (P3) is still unidentifiable, and thus information for image recognition such as person detection and situation recognition cannot be extracted.

[0084] Accordingly, even if a third party acquires a plurality of output images and their original images (i.e., input images) output through the mirrorless camera (100), it is difficult to generate the plurality of restoration models by artificial intelligence learning them, so the security function can be strengthened. This means that the security function can be further strengthened as the number of the plurality of PSFs increases and as the PSF identification information is encrypted and included in the output image. In the case where the PSF identification information is encrypted and included in the output image, the electronic device may further require a step of decrypting the extracted PSF identification information.

[0085] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include media implemented in the form of carrier waves (e.g., transmission via the Internet).

[0086] Accordingly, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. Variable optical modulation mask; image sensor; and A memory storing data on multiple PSFs (Point Spread Functions); and Selecting one of the above plurality of PSFs and controlling the variable optical modulation mask to have an optical modulation pattern of the selected PSF, A lensless camera including a processor that performs image processing on an input image that is optically modulated according to the optical modulation pattern of the selected PSF to generate an output image, wherein the output image includes PSF identification information corresponding to the selected PSF.

2. In the first paragraph, the processor, A lensless camera characterized in that the optical modulation pattern of the variable optical modulation mask is controlled to be periodically changed to another optical modulation pattern among the plurality of PSFs.

3. In paragraph 1, A lensless camera, wherein the variable optical modulation mask comprises a transmissive display element.

4. In paragraph 1, A lensless camera, characterized in that the PSF identification information is encrypted and included in the output image using steganography.

5. Video input section; A memory storing multiple restoration models corresponding to each of multiple PSFs; and Receives video footage taken by a lensless camera through the above video input unit, An electronic device comprising a control unit that controls restoration of the photographed image by using a restoration model corresponding to PSF identification information extracted from the photographed image among the plurality of restoration models.

6. In paragraph 5, An electronic device characterized in that each of the plurality of restoration models is generated by artificial intelligence learning of input / output image learning data prepared in advance according to each of the plurality of PSFs.

7. A step of selecting one of a plurality of PSFs and controlling the variable optical modulation mask to have an optical modulation pattern of the selected PSF; A step of generating an output image by performing image processing on an input image that has been optically modulated according to the optical modulation pattern of the selected PSF, wherein the output image includes PSF identification information corresponding to the selected PSF; A method for controlling a lensless camera, comprising: a step of restoring an output image using a restoration model corresponding to PSF identification information extracted from an output image among a plurality of restoration models; 8. In paragraph 7, A method for controlling a lensless camera, comprising: a step of periodically changing the optical modulation pattern of the variable optical modulation mask to another optical modulation pattern among the plurality of PSFs.

9. In paragraph 7, A method for controlling a lensless camera, characterized in that the PSF identification information is encrypted and included in the output image using steganography.

10. In paragraph 7, A method for controlling a lensless camera, characterized in that each of the plurality of restoration models is generated by artificial intelligence learning of input / output image learning data prepared in advance according to each of the plurality of PSFs.

Images