Multi-pinhole camera and image recognition system

Abstract

The gap between a pinhole (301aa) and a pinhole (301ab) is set to a first gap (L1) where when a subject (302) located at a distance less than a prescribed distance (D) from a multi-pinhole camera (301) has been imaged, the degree of overlap of subject images (302a, 302b) imaged via the pinholes (301aa, 301ab) is within a prescribed range. The gap between the pinhole (301ab) and a pinhole (301ac) is set to a second gap (L2), which is narrower than the first gap (L1), where when the subject (302) located at a distance greater than or equal to the prescribed distance (D) from the multi-pinhole camera (301) has been imaged, the degree of overlap of subject images (302b, 302c) imaged via the pinholes (301ab, 301ac) is within a prescribed range.

Description

[Technical field]

[0001] The present disclosure relates to an image identification system, particularly in an environment where privacy protection is required, such as in a home or indoors, and a multi-pinhole camera used therein. [Background technology]

[0002] The following Patent Document 1 discloses an image identification system in which a computationally captured image is input to a classifier, and the classifier identifies an object included in the computationally captured image using a trained identification model. The following Patent Document 1 also discloses a light field camera using a multi-pinhole mask in which multiple pinholes are formed, as an example of a camera for capturing a computationally captured image.

[0003] A computationally captured image is an image that is difficult for humans to visually recognize due to intentional blurring caused by the superposition of multiple images with different viewpoints, or by the effect of making it difficult to focus on a subject image by not using a lens, etc. Therefore, it is preferable to use computationally captured images for building an image recognition system in an environment where privacy protection is required, particularly in a home or indoors.

[0004] However, in the above-mentioned Patent Document 1, there is no detailed consideration of how to set the intervals between the multiple pinholes in the multi-pinhole mask. Therefore, depending on the distance between the camera and the subject, the desired blur may not be obtained, and the privacy of the subject may not be protected. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 054092 Summary of the Invention

[0006] The present disclosure aims to provide a technology in an image recognition system that can protect the privacy of a subject regardless of the distance between the camera and the subject.

[0007] A multi-pinhole camera according to one embodiment of the present disclosure comprises an image sensor and a mask arranged between a light receiving surface of the image sensor and a subject, the mask having a mask pattern in which a plurality of pinholes are formed, including a first pinhole, a second pinhole adjacent to the first pinhole, and a third pinhole adjacent to the second pinhole, wherein the distance between the first pinhole and the second pinhole is set to a first distance, the first distance being a distance at which, when an image of a subject located at a distance less than a predetermined distance from the multi-pinhole camera is captured by the multi-pinhole camera, the degree of overlap of two subject images captured through the first pinhole and the second pinhole is within a predetermined range, and the distance between the second pinhole and the third pinhole is set to a second distance narrower than the first distance, the second distance being a distance at which, when an image of a subject located at a distance equal to or greater than the predetermined distance from the multi-pinhole camera is captured by the multi-pinhole camera, the degree of overlap of two subject images captured through the second pinhole and the third pinhole is within the predetermined range. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of an image identification system according to an embodiment. [Diagram 2] 1 is a flowchart showing a procedure of main processes of the image identification system. [Diagram 3] FIG. 1 is a diagram illustrating a schematic structure of a lensless multi-pinhole camera. [Figure 4A] FIG. 2 is a schematic diagram showing the configuration of a multi-pinhole mask. [Figure 4B] FIG. 2 is a diagram showing an example of an image captured by a multi-pinhole camera. [Figure 4C]FIG. 2 is a diagram showing an example of an image captured by a multi-pinhole camera. [Figure 5A] FIG. 2 is a schematic diagram showing the configuration of a multi-pinhole mask. [Figure 5B] FIG. 2 is a diagram showing an example of an image captured by a multi-pinhole camera. [Figure 5C] FIG. 2 is a diagram showing an example of an image captured by a multi-pinhole camera. [Figure 6A] FIG. 2 is a schematic diagram showing the configuration of a multi-pinhole mask. [Figure 6B] FIG. 2 is a diagram showing an example of an image captured by a multi-pinhole camera. [Figure 6C] FIG. 2 is a diagram showing an example of an image captured by a multi-pinhole camera. [Figure 7] FIG. 11 is a diagram showing a predetermined distance between a multi-pinhole camera and a subject. [Figure 8] 4 is a flowchart showing a procedure of main processes performed by the learning device. [Figure 9] FIG. 13 is a schematic diagram showing the structure of a multi-pinhole camera according to a first modified example. [Figure 10] FIG. 11 is a schematic diagram showing the structure of a multi-pinhole camera according to a second modified example. [Figure 11] FIG. 13 is a schematic diagram showing a structure of a multi-pinhole camera according to a first example of a third modified example. [Figure 12] FIG. 13 is a schematic diagram showing a structure of a multi-pinhole camera according to a second example of the third modified example. [Figure 13A] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 13B] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 13C] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 13D] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 14A] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 14B] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 14C] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 14D] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 15] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 16] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 17] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 18A] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 18B] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 18C] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 18D] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 18E] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 18F] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 19A] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 19B] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. [Figure 19C] FIG. 13 is a schematic diagram showing a configuration of a multi-pinhole camera according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (Findings on which this disclosure is based) Various recognition technologies are important in the home or indoors, such as recognizing the behavior of people in the environment and recognizing the person who operates the equipment. In recent years, a technology called deep learning has been attracting attention for object recognition. Deep learning is a machine learning method that uses a multi-layered neural network, and by using a large amount of training data, it is possible to achieve more accurate recognition performance than conventional methods. Image information is particularly effective in such object recognition. Various methods have been proposed that use a camera as an input device and perform deep learning using image information as input to significantly improve conventional object recognition capabilities.

[0010] However, placing cameras in homes and other locations poses the problem that privacy may be violated if the captured images are leaked to the outside due to hacking, etc. Therefore, measures are needed to protect the privacy of the subjects even if the captured images are leaked to the outside.

[0011] A computationally captured image captured by a multi-pinhole camera or the like is an image that is difficult for humans to visually recognize due to intentionally created blurring caused by the superposition of multiple images from different viewpoints, or by the effect of the subject image being difficult to focus on due to the lack of a lens, etc. Therefore, it is preferable to use computationally captured images to build an image recognition system in an environment where privacy protection is required, particularly in the home or indoors.

[0012] In the image identification system disclosed in the above Patent Document 1, a light field camera having a multi-pinhole mask is used to capture an image of a target area, and the calculated captured image obtained by the image capture is input to a classifier. As a result, the classifier uses a trained identification model to identify objects contained in the calculated captured image. In this way, by capturing an image of a target area using a light field camera capturing a calculated captured image, the subject's privacy can be protected, even if the captured image is leaked to the outside, because the calculated captured image is difficult for humans to visually recognize.

[0013] However, in the above Patent Document 1, the setting of the interval between the multiple pinholes in the multi-pinhole mask is not considered in detail. Therefore, depending on the distance between the camera and the subject, the desired difficulty of visual recognition may not be obtained, and the privacy of the subject may not be protected. For example, if the interval between the pinholes is too wide, when the distance between the camera and the subject is relatively far, the amount of deviation becomes too large for a small subject image, and the subject's face is recognized without the multiple subject images superimposing. On the other hand, if the interval between the pinholes is too narrow, when the distance between the camera and the subject is relatively close, the amount of deviation becomes too small for a large subject image, and the subject's face is recognized.

[0014] In order to solve this problem, the inventor discovered that by using a mask pattern with at least three pinholes spaced apart and setting the spacing between each pinhole so that the degree of overlap of the subject image is within a specified range, it is possible to protect the privacy of the subject regardless of the distance between the camera and the subject, and thus came up with the present disclosure.

[0015] Next, each aspect of the present disclosure will be described.

[0016] A multi-pinhole camera according to one embodiment of the present disclosure comprises an image sensor and a mask arranged between a light receiving surface of the image sensor and a subject, the mask having a mask pattern in which a plurality of pinholes are formed, including a first pinhole, a second pinhole adjacent to the first pinhole, and a third pinhole adjacent to the second pinhole, wherein the distance between the first pinhole and the second pinhole is set to a first distance, the first distance being a distance at which the overlapping degree of two subject images captured through the first pinhole and the second pinhole falls within a predetermined range when the multi-pinhole camera captures an image of a subject located at a distance less than a predetermined distance from the multi-pinhole camera, and the distance between the second pinhole and the third pinhole is set to a second distance narrower than the first distance, the second distance being a distance at which the overlapping degree of two subject images captured through the second pinhole and the third pinhole falls within the predetermined range when the multi-pinhole camera captures an image of a subject located at a distance equal to or greater than the predetermined distance from the multi-pinhole camera.

[0017] According to this configuration, when the subject is located at a distance less than a predetermined distance from the multi-pinhole camera, the overlapping degree of the two subject images captured through the first pinhole and the second pinhole falls within a predetermined range, so that the multiple subject images are visually incapable of being individually recognized by humans due to the overlapping images, thereby protecting the privacy of the subject. Also, when the subject is located at a distance equal to or greater than a predetermined distance from the multi-pinhole camera, the overlapping degree of the two subject images captured through the second pinhole and the third pinhole falls within a predetermined range, so that the multiple subject images are visually incapable of being individually recognized by humans due to the overlapping images, thereby protecting the privacy of the subject. As a result, it is possible to protect the privacy of the subject regardless of the distance between the multi-pinhole camera and the subject.

[0018] In the above aspect, when the subject located at a distance greater than or equal to the predetermined distance from the multi-pinhole camera is imaged by the multi-pinhole camera, the degree of overlap of the two subject images imaged through the first pinhole and the second pinhole is outside the predetermined range, and when the subject located at a distance less than the predetermined distance from the multi-pinhole camera is imaged by the multi-pinhole camera, the degree of overlap of the two subject images imaged through the second pinhole and the third pinhole is outside the predetermined range.

[0019] According to this configuration, even if the privacy of the subject cannot be completely protected by only one of the first interval and the second interval, by mixing the first interval and the second interval within the same mask, it is possible to protect the privacy of the subject regardless of the distance between the multi-pinhole camera and the subject.

[0020] In the above aspect, the multi-pinhole camera may not have an optical system that forms an image of light from a subject on the image sensor.

[0021] According to this configuration, since the multi-pinhole camera does not have an optical system that focuses light from the subject on the image sensor, it is possible to intentionally create a blur in the image captured by the multi-pinhole camera. As a result, it becomes more difficult for humans to visually identify the subject in the captured image, and it is possible to further enhance the effect of protecting the privacy of the subject.

[0022] In the above aspect, the first pinhole, the second pinhole, and the third pinhole may have opening areas different from one another.

[0023] According to this configuration, the aperture area of ​​each pinhole is different, so that the degree of blur of each subject image is also different. The mixture of multiple subject images with different degrees of blur makes the captured image more complicated. As a result, it becomes more difficult for a human to visually identify the subjects included in the captured image, so that the effect of protecting the privacy of the subjects can be further improved.

[0024] In the above aspect, when the mask is divided into a plurality of regions, in the mask pattern, each of the plurality of regions may include a pinhole set consisting of the first pinhole and the second pinhole, and a pinhole set consisting of the third pinhole and the second pinhole.

[0025] With this configuration, even if the angle between the front direction of the multi-pinhole camera and the position of the subject is large, the light from the subject can reach the image sensor through the pinhole set closer to the subject, which makes it possible to expand the angle of view at which multiple images can be realized.

[0026] In the above aspect, in the mask pattern, the number of pinhole sets having the first pinhole and the second pinhole may be greater than the number of pinhole sets having the third pinhole and the second pinhole.

[0027] According to this configuration, when the subject is located at a distance less than a predetermined distance from the multi-pinhole camera, the number of effective overlapping images can be increased. As a result, it becomes more difficult for humans to visually identify the subject included in the captured image, so that the effect of protecting the privacy of the subject can be further improved. This is particularly effective for indoor use where the distance between the multi-pinhole camera and the subject is likely to be close.

[0028] In the above aspect, in the mask pattern, the number of pinhole sets each having the third pinhole and the second pinhole may be greater than the number of pinhole sets each having the first pinhole and the second pinhole.

[0029] According to this configuration, when the subject is located at a distance equal to or greater than a predetermined distance from the multi-pinhole camera, the number of effective overlapping images can be increased. As a result, it becomes more difficult for humans to visually identify the subject in the captured image, so that the effect of protecting the privacy of the subject can be further improved. This is particularly effective for outdoor use, where the distance between the multi-pinhole camera and the subject is likely to be large.

[0030] An image recognition system according to one embodiment of the present disclosure comprises a multi-pinhole camera according to the above embodiment, a recognition unit that recognizes an image captured by the multi-pinhole camera based on an image recognition model, and an output unit that outputs the recognition result by the recognition unit.

[0031] According to this configuration, when the subject is located at a distance less than a predetermined distance from the multi-pinhole camera, the overlapping degree of the two subject images captured through the first pinhole and the second pinhole falls within a predetermined range, so that the multiple subject images are visually incapable of being individually recognized by humans due to the overlapping images, thereby protecting the privacy of the subject. Also, when the subject is located at a distance equal to or greater than a predetermined distance from the multi-pinhole camera, the overlapping degree of the two subject images captured through the second pinhole and the third pinhole falls within a predetermined range, so that the multiple subject images are visually incapable of being individually recognized by humans due to the overlapping images, thereby protecting the privacy of the subject. As a result, it is possible to protect the privacy of the subject regardless of the distance between the multi-pinhole camera and the subject.

[0032] The present disclosure can be realized as a computer program for causing a computer to execute each characteristic configuration included in such a method, or as an apparatus or system that operates based on this computer program. Furthermore, it goes without saying that such a computer program can be distributed as a computer-readable non-volatile recording medium such as a CD-ROM, or distributed via a communication network such as the Internet.

[0033] Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim showing a top concept are described as optional components. Furthermore, in all embodiments, the contents of each can be combined.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In addition, elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0035] (Embodiments of the present disclosure) FIG. 1 is a schematic diagram showing a configuration of an image recognition system 10 according to an embodiment of the present disclosure. The image recognition system 10 includes a learning device 20 and a recognition device 30. The recognition device 30 includes a multi-pinhole camera 301, a recognition unit 106, and an output unit 107. The recognition unit 106 includes a processor such as a CPU and a memory such as a semiconductor memory. The output unit 107 includes a display device or a speaker. The learning device 20 includes a learning database 102, a PSF information acquisition unit 103, a database correction unit 104, and a learning unit 105. The learning database 102 includes a storage unit such as an HDD, an SSD, or a semiconductor memory. The PSF information acquisition unit 103, the database correction unit 104, and the learning unit 105 include processors such as a CPU.

[0036] FIG. 2 is a flowchart showing the procedure of the main processing of the image identification system 10. The flowchart shows the flow of the image identification processing by the identification device 30. First, the multi-pinhole camera 301 captures an image of a target area in which a subject to be image-identified is located, and inputs a calculated captured image obtained by the capture to the identification unit 106 (step S101). Next, the identification unit 106 uses a trained image identification model to identify the calculated captured image (step S102). This image identification model is an image identification model created by learning by the learning device 20. Next, the output unit 107 outputs the result of the identification by the identification unit 106. Details of the processing of each step will be described later.

[0037] Unlike a normal camera that captures a normal image that is easy for humans to visually recognize, the multi-pinhole camera 301 captures a computationally captured image that is an image that is difficult for humans to visually recognize due to overlapping images. A computationally captured image is an image in which a person cannot recognize a subject even if he or she looks at the captured image itself, but by performing image processing on the captured computationally captured image, an image that can be recognized by a person or identified by the recognition unit 106 can be generated.

[0038] FIG. 3 is a diagram showing a schematic structure of a lensless multi-pinhole camera 301. The multi-pinhole camera 301 shown in FIG. 3 has a multi-pinhole mask 301a and an image sensor 301b such as a CMOS. The multi-pinhole mask 301a is disposed between the image sensor 301b and the subject 302, at a certain distance from the light receiving surface of the image sensor 301b. A cover glass for IR cut and anti-reflection may be disposed between the multi-pinhole mask 301a and the image sensor 301b. The cover glass and the image sensor 301b may be separated or in contact with each other. The cover glass and the multi-pinhole mask 301a may be separated or in contact with each other. The image sensor 301b, the cover glass, and the multi-pinhole mask 301a are supported by a housing having a holder or pillar formed therein for holding them.

[0039] The multi-pinhole mask 301a has a plurality of pinholes 301aa to 301ac arranged linearly in this order. The distance between the pinhole 301aa and the pinhole 301ab is L1, and the distance between the pinhole 301ab and the pinhole 301ac is L2 (<L1). In the example of the present embodiment, the shapes and opening areas of the pinholes 301aa to 301ac are equal to each other. The plurality of pinholes 301aa to 301ac are also called multi-pinholes. The image sensor 301b acquires an image of the subject 302 through each of the pinholes 301aa to 301ac. The image acquired through the pinhole is called a pinhole image. Note that the plurality of pinholes 301aa to 301ac do not necessarily have to be arranged linearly. For example, they may be arranged at the vertices of a scalene triangle.

[0040] Since the pinhole images of the subject 302 are different depending on the positions and sizes of the respective pinholes 301aa to 301ac, the image sensor 301b acquires a superimposed image in a state where a plurality of pinhole images are slightly shifted and overlapped (that is, a multiple image). The positional relationship of the plurality of pinholes 301aa to 301ac affects the positional relationship of the plurality of pinhole images projected on the image sensor 301b (that is, the degree of overlap of the multiple images), and the sizes of the pinholes 301aa to 301ac affect the degree of blurring of the pinhole images.

[0041] By using the multi-pinhole mask 301a, it is possible to obtain a plurality of pinhole images with different positions and degrees of blurring by superimposing them. In other words, it is possible to obtain a calculated capture image that is difficult for humans to visually recognize due to the influence of superimposing a plurality of images with different viewpoints, or the difficulty of focusing the subject image due to the absence of a lens. Therefore, the captured image is a multiple image and a blurred image, and due to these influences, an image in which the privacy of the subject 302 is protected can be obtained. In addition, by changing the number, position, and size of each pinhole, it is possible to obtain images with different degrees of superimposition of multiple images and different degrees of blurring. In other words, the multi-pinhole mask 301a may be structured so that it can be easily attached and detached by the user, and a plurality of types of multi-pinhole masks 301a with different mask patterns may be prepared in advance, allowing the user to freely replace the multi-pinhole mask 301a to be used.

[0042] In addition to replacing the mask, such mask changes can also be made: The user can freely rotate the mask that is attached in front of the image sensor. The user drills holes at any point on the plate attached in front of the image sensor. By using a liquid crystal mask that utilizes a spatial light modulator, the transmittance at each position in the mask can be set arbitrarily. - The mask is made of elastic material such as rubber, and the position and size of the holes can be changed by physically deforming the mask through the application of external force. These modifications will be described below in order.

[0043] 13A to 13D are schematic diagrams showing the configuration of a multi-pinhole camera 301 in which a user can rotate a mask at will. FIG. 13A shows an overview of the multi-pinhole camera 301 in which a user can rotate a mask at will, and FIG. 13B shows a schematic diagram of the cross section. The multi-pinhole camera 301 has a multi-pinhole mask 301a that can rotate with respect to its housing 401, and a gripping part 402 is connected to the multi-pinhole mask 301a. A user can fix or rotate the multi-pinhole mask 301a with respect to the housing 401 by gripping and operating the gripping part 402. Such a mechanism may be such that a screw is provided on the gripping part 402, and the multi-pinhole mask 301a is fixed by tightening the screw, and the multi-pinhole mask 301a can be rotated by loosening the screw. FIG. 13C and FIG. 13D show schematic diagrams in which the multi-pinhole mask 301a rotates 90 degrees when the gripping part 402 is rotated 90 degrees. In this manner, by the user moving the grip portion 402, the multi-pinhole mask 301a can be rotated.

[0044] In addition, in the multi-pinhole camera 301 in which the user can rotate the mask as desired, the multi-pinhole mask 301a may have pinholes arranged asymmetrically with respect to rotation, as shown in Fig. 13C. In this way, it is possible to realize various multi-pinhole patterns by the user rotating the mask.

[0045] Of course, the configuration of the multi-pinhole camera 301 in which the user can rotate the mask at will may not have the grip part 402. FIGS. 14A and 14B are schematic diagrams showing another configuration example of the multi-pinhole camera 301 in which the user can rotate the mask at will. FIG. 14A shows an overview of another configuration example of the multi-pinhole camera 301 in which the user can rotate the mask at will, and FIG. 14B shows a schematic diagram of the cross section. The multi-pinhole mask 301a is fixed to the lens barrel 411. The image sensor 301b is installed in another lens barrel 412, and the lens barrel 411 and the lens barrel 412 are in a rotatable state by a screw configuration. That is, the lens barrel 412 is located outside the lens barrel 411, and a male screw is cut on the outside of the lens barrel 411, which is the joint, and a female screw is cut on the inside of the lens barrel 412. Also, a fixing tool 413 is first attached to the male screw of the lens barrel 411, and then the lens barrel 412 is attached. The fixture 413 is also threaded with a female screw, like the lens barrel 412. With this configuration, when the lens barrel 411 is screwed into the lens barrel 412, the screwing depth changes depending on the screwing position of the fixture 413 into the lens barrel 411, and the rotation angle of the multi-pinhole camera 301 can be changed.

[0046] 14C and 14D are schematic diagrams showing that the screwing depth changes depending on the screwing position of the fixing tool 413 into the lens barrel 411, and the rotation angle of the multi-pinhole camera 301 changes. FIG. 14C is a schematic diagram of the case where the fixing tool 413 is screwed into the lens barrel 411 to the back, and FIG. 14D is a schematic diagram of the case where the fixing tool 413 is screwed into the lens barrel 411 only halfway. As shown in FIG. 14C, when the fixing tool 413 is screwed into the lens barrel 411 to the back, the lens barrel 412 can be screwed into the lens barrel 411 to the back. On the other hand, as shown in FIG. 14D, when the fixing tool 413 is screwed into the lens barrel 411 only halfway, the lens barrel 412 can be screwed into the lens barrel 411 only halfway. Therefore, depending on the screwing position of the fixing tool 413 into the lens barrel 411, the screwing depth changes, and the rotation angle of the multi-pinhole mask 301a can be changed.

[0047] Fig. 15 is a schematic diagram of a cross section of a multi-pinhole camera 301 in which a user can make holes at any desired positions in a mask 301ab attached in front of an image sensor 301b. In Fig. 15, the same components as those in Fig. 13 are given the same reference numerals, and the description thereof will be omitted. Initially, there are no pinholes in the mask 301ab. A multi-pinhole mask of any shape can be created by the user making multiple holes at any desired positions in this mask 301ab using a needle or the like.

[0048] <Modification in which the transmittance at each position in the mask is set arbitrarily using a spatial light modulator> FIG. 16 is a schematic diagram of a cross section of a multi-pinhole camera 301 configured to arbitrarily set the transmittance at each position in the mask using a spatial light modulator 420. In FIG. 16, the same components as those in FIG. 15 are given the same reference numerals, and the description will be omitted. The spatial light modulator 420 is made of liquid crystal or the like, and can change the transmittance for each pixel. This spatial light modulator 420 functions as a multi-pinhole mask. The change in the transmittance can be controlled by a spatial light modulator control unit (not shown). Therefore, various mask patterns (multi-pinhole patterns) can be realized by the user selecting an arbitrary pattern from a plurality of transmittance patterns prepared in advance.

[0049] <Modification in which the mask is deformed by application of an external force> 17, 18A to 18F are schematic diagrams of a cross section of a multi-pinhole camera 301 configured to deform a mask by applying an external force. In FIG. 17, the same components as those in FIG. 15 are given the same reference numerals, and the description is omitted. The multi-pinhole mask 301ac is configured of a plurality of masks 301a1, 301a2, and 301a3, and each mask has a driving unit (not shown) that applies an external force independently. FIGS. 18A to 18C are schematic diagrams for explaining three masks 301a1, 301a2, and 301a3 that configure the multi-pinhole mask 301ac. Here, each mask has a shape that combines a sector shape and a ring. Of course, this configuration is an example, and the shape is not limited to a sector shape, and the number of masks is not limited to three. Each mask has one or more pinholes formed therein. It is not necessary that a pinhole is formed in the mask. Two pinholes 301aa1 and 301aa2 are formed in mask 301a1, one pinhole 301aa3 is formed in mask 301a2, and two pinholes 301aa4 and 301aa5 are formed in mask 301a3. By moving these three masks 301a1 to 301a3 by applying an external force, various multi-pinhole patterns can be created.

[0050] 18D-18F show three types of multi-pinhole masks 301ac consisting of three masks 301a1-301a3. Each driving unit (not shown) moves each mask 301a1-301a3 in a different manner, so that a mask having five pinholes is configured in FIGS. 18D and 18E, and a mask having four pinholes is configured in FIG. 18F. The driving units of such masks can be realized by using ultrasonic motors or linear motors that are widely used in autofocus and the like. In this way, the number and positions of the pinholes in the multi-pinhole mask 301ac can be changed by applying an external force.

[0051] Of course, the multi-pinhole mask may be designed to change not only the number and positions of the pinholes, but also the size of the pinholes. Figures 19A to 19C are schematic diagrams for explaining the configuration of a multi-pinhole mask 301ad in a multi-pinhole camera 301 configured to deform the mask by application of an external force. The multi-pinhole mask 301ad has a plurality of pinholes, is made of an elastic material, and has four driving units 421 to 424 that can independently control the four corners. Of course, the number of driving units does not have to be four. By moving each of the driving units 421 to 424, the positions and sizes of the pinholes in the multi-pinhole mask 301ad can be changed.

[0052] FIG. 19B is a schematic diagram showing a state in which the driving units 421-424 are moved in the same direction. In this figure, the direction of the arrows shown on the driving units 421-424 indicates the driving direction of each driving unit. In this case, the multi-pinhole mask 301ad moves parallel to the driving direction of the driving units. On the other hand, FIG. 19C is a schematic diagram showing a state in which the driving units 421-424 are moved in an outward direction from the center of the multi-pinhole mask 301ad. In this case, the multi-pinhole mask 301ad is stretched according to its elasticity, so the size of the pinholes becomes larger. Such driving units 421-424 can be realized by using an ultrasonic motor or a linear motor that is widely used in autofocus and the like. In this way, the position and size of the pinholes in the multi-pinhole mask 301ac can be changed by applying an external force.

[0053] Furthermore, even with the same subject and the same multi-pinhole mask, the state of overlapping images differs depending on the distance between the multi-pinhole camera 301 and the subject 302. Therefore, in the multi-pinhole camera 301 according to this embodiment, by varying the intervals L1, L2 between the pinholes, an image having effective overlapping images is captured regardless of the distance between the subject 302 and the multi-pinhole camera 301.

[0054] The effectiveness of making the intervals L1, L2 between the pinholes different will be described.

[0055] Fig. 4A is a schematic diagram showing the configuration of multi-pinhole mask 301a when the intervals between pinholes are standardized to narrow interval L2. Fig. 4B is a diagram showing a captured image when subject 302 located far from multi-pinhole camera 301 is captured. Fig. 4C is a diagram showing a captured image when subject 302 located near multi-pinhole camera 301 is captured. Images captured through pinholes 301aa to 301ac become subject images 302a to 302c, respectively.

[0056] When the intervals between the pinholes are standardized to a narrow interval L2 as shown in FIG. 4A, the amount of deviation between the subject images 302a to 302c captured through the pinholes 301aa to 301ac becomes small.

[0057] Therefore, when the subject 302 is located far away from the multi-pinhole camera 301 and the subject image is small, the subject image captured through each pinhole is effectively superimposed on the subject image captured through the adjacent pinhole, so that an image in which the privacy of the subject 302 is protected can be captured by effectively superimposing the overlapping images, as shown in FIG 4B.

[0058] On the other hand, if the subject 302 is located near the multi-pinhole camera 301 and therefore the subject image is large, the subject image captured through each pinhole will be slightly overlapped with the subject image captured through the adjacent pinhole. However, since the amount of shift is too small compared to the size of the subject image, this does not result in an effective overlapping image superposition, as shown in Fig. 4C, and the privacy of the subject 302 cannot be protected.

[0059] Fig. 5A is a schematic diagram showing the configuration of a multi-pinhole mask 301a in which the intervals between pinholes are unified to a wide interval L1. Fig. 5B is a diagram showing a captured image when an object 302 located far from the multi-pinhole camera 301 is captured. Fig. 5C is a diagram showing a captured image when an object 302 located near the multi-pinhole camera 301 is captured.

[0060] When the intervals between the pinholes are unified to a wide interval L1 as shown in FIG. 5A, the amount of deviation between the subject images 302a to 302c captured through the pinholes 301aa to 301ac increases.

[0061] Therefore, when the subject 302 is located near the multi-pinhole camera 301 and the subject image is large, the subject image captured through each pinhole is effectively superimposed on the subject image captured through the adjacent pinhole, so that an image can be captured in which the privacy of the subject 302 is protected by effective superimposition of multiple images, as shown in Fig. 5C.

[0062] On the other hand, when the subject 302 is located far away from the multi-pinhole camera 301 and the subject image is small, the amount of deviation is too large compared to the size of the subject image, so the subject images captured through each pinhole do not overlap, and the privacy of the subject 302 cannot be protected.

[0063] In this way, if the spacing between pinholes is standardized to L1 or L2, it is difficult to capture privacy-protected images by effectively superimposing multiple images for all subjects 302 that are at different distances from the multi-pinhole camera 301.

[0064] The multi-pinhole camera 301 according to this embodiment has a multi-pinhole mask 301a having a plurality of pinholes 301aa to 301ac, and is characterized in that the intervals L1, L2 between the pinholes are different.

[0065] FIG. 6A is a schematic diagram showing the configuration of the multi-pinhole mask 301a according to the present embodiment. The distance between the pinhole 301aa and the pinhole 301ab is L1, and the distance between the pinhole 301ab and the pinhole 301ac is L2 (<L1). FIG. 6A shows an example in which three pinholes 301aa to 301ac are formed, but the number of pinholes may be three or more. FIG. 6B is a diagram showing an imaging image when the subject 302 located far from the multi-pinhole camera 301 is imaged. FIG. 6C is a diagram showing an imaging image when the subject 302 located near the multi-pinhole camera 301 is imaged.

[0066] As shown in FIG. 6A, the distances L1 and L2 between the pinholes are different. Therefore, the subject images 302a to 302c captured through each of the pinholes 301aa to 301ac are superimposed with different amounts of displacement. Therefore, even when the subject image is small because the subject 302 is located far from the multi-pinhole camera 301, or even when the subject image is large because the subject 302 is located near the multi-pinhole camera 301, the subject images captured through each pinhole are superimposed on the subject images captured through the adjacent pinholes. As a result, as shown in FIGS. 6B and 6C, an image that protects the privacy of the subject 302 can be captured by the effective superposition of multiple images.

[0067] Specifically, when the subject 302 is located at a distance of a predetermined distance D or more from the multi-pinhole camera 301 and the subject image is small (FIG. 6B), the subject images 302b and 302c captured through the pinholes 301ab and 301ac with a narrow gap L2 are effectively superimposed with a small amount of deviation. When the subject 302 is located at a distance less than the predetermined distance D from the multi-pinhole camera 301 and the subject image is large (FIG. 6C), the subject images 302a and 302b captured through the pinholes 301aa and 301ab with a wide gap L1 are effectively superimposed with a large amount of deviation. This makes it possible to capture an image in which the privacy of the subject 302 is protected by effective superimposition of overlapping images. Note that "effective superimposition" means that the degree of overlap (degree of overlap in area) of the two subject images captured through two adjacent pinholes is within a predetermined allowable range. For example, the lower limit of the allowable range is 30% and the upper limit is 90%. In other words, if the overlap rate of two subject images obtained through two adjacent pinholes is between 30% and 90%, a third party will not be able to identify the subjects individually even when looking at the captured image.

[0068] 7 is a diagram showing a predetermined distance D between the multi-pinhole camera 301 and the subject 302. The predetermined distance D is set according to the intervals L1 and L2, the distance between the image sensor 301b and the multi-pinhole mask 301a, the sensor size and pixel pitch of the image sensor 301b, and the like, and is set to, for example, 5 m. In this example, when the subject 302 is located at a distance of less than 5 m from the multi-pinhole camera 301, the wide interval L1 effectively functions to effectively superimpose the subject images 302a and 302b. On the other hand, when the subject 302 is located at a distance of 5 m or more from the multi-pinhole camera 301, the narrow interval L2 effectively functions to effectively superimpose the subject images 302b and 302c.

[0069] When the subject 302 located at a distance equal to or greater than the predetermined distance D from the multi-pinhole camera 301 is imaged by the multi-pinhole camera 301 (FIG. 6B), the overlapping degree of the two subject images 302a, 302b imaged through the pinholes 301aa, 301ab with a wide spacing L1 falls outside the above-mentioned predetermined range (lower limit value 30%). Similarly, when the subject 302 located at a distance less than the predetermined distance D from the multi-pinhole camera 301 is imaged by the multi-pinhole camera 301 (FIG. 6C), the overlapping degree of the two subject images 302b, 302c imaged through the pinholes 301ab, 301ac with a narrow spacing L2 falls outside the above-mentioned predetermined range (upper limit value 90% or more). Even if the privacy of the subject 302 cannot be completely protected by only one of the distances L1 and L2 in this way, by mixing the distances L1 and L2 within the same multi-pinhole mask 301a, it is possible to protect the privacy of the subject 302 regardless of the distance between the multi-pinhole camera 301 and the subject 302.

[0070] 3, it is preferable that the multi-pinhole camera 301 does not have an optical system (lens, prism, mirror, etc.) for forming an image of the light from the subject 302 on the image sensor 301b. By omitting the optical system, the camera can be made smaller and lighter, the cost can be reduced, and the design can be improved, and a blur can be intentionally created in the image captured by the camera. As a result, it becomes more difficult for a human being to visually identify the subject 302 included in the captured image, so that the effect of protecting the privacy of the subject 302 can be further improved.

[0071] The image recognition system 10 will be described with reference to Fig. 1. The recognition unit 106 uses an image recognition model that is a learning result of the learning device 20 to recognize category information of subjects, such as people (including their behavior and facial expressions), cars, bicycles, or traffic lights, contained in an image of a target area captured by a multi-pinhole camera 301, and position information of each subject. Machine learning such as deep learning using a multi-layer neural network may be used for learning to create the image recognition model.

[0072] The output unit 107 outputs the result of the identification by the identification unit 106. This may have an interface unit and present the identification result to the user by image, text, sound, or the like, or may have a device control unit and change the control method depending on the identification result.

[0073] The learning device 20 includes a learning database 102, a PSF information acquisition unit 103, a database correction unit 104, and a learning unit 105. The learning device 20 performs learning to create an image recognition model to be used by the recognition unit 106 in correspondence with PSF information related to a computational imaging camera 101 that is actually used to capture an image of a target area.

[0074] FIG. 8 is a flowchart showing the procedure of main processes performed by the learning device 20 of the image recognition system 10.

[0075] First, the PSF information acquisition unit 103 acquires PSF information, which is information that indicates the aspect of the calculated captured image, and what kind of calculated captured image is captured by the calculated captured camera 101 (step S201). This may be performed by having the multi-pinhole camera 301 include a transmitter and the PSF information acquisition unit 103 include a receiver, and the PSF information may be exchanged by wire or wirelessly, or the PSF information acquisition unit 103 may include an interface, and the user may input the PSF information to the PSF information acquisition unit 103.

[0076] The PSF (Point Spread Function) is a transfer function of a camera such as a multi-pinhole camera or a coded aperture camera, and is expressed by the following relationship:

[0077] y=k*x

[0078] Here, y is a blurred calculated image captured by the multi-pinhole camera 301, k is a PSF, and x is a normal image of the scene captured by a normal camera without blur. Also, * is a convolution operator.

[0079] The PSF can be obtained by capturing a point light source with the multi-pinhole camera 301. This can be seen from the fact that the PSF corresponds to the impulse response of the camera. In other words, the captured image of the point light source obtained by capturing the point light source with the multi-pinhole camera 301 is itself the PSF as captured image information of the multi-pinhole camera 301. Here, as the captured image of the point light source, it is desirable to use a difference image between a captured image including a point light source in a lit state and a captured image including the point light source in an unlit state.

[0080] Next, the database correction unit 104 acquires a normal image without blur contained in the training database 102, and the learning unit 105 acquires annotation information contained in the training database 102 (step S202).

[0081] Next, the database correction unit 104 corrects the learning database 102 by using the PSF information acquired by the PSF information acquisition unit 103 (step S203). For example, when the identification unit 106 identifies the behavior of a person in an environment, the learning database 102 holds a plurality of normal images taken by a normal camera without blur, and annotation information (correct answer label) to be assigned to each image, which indicates the position and behavior of the person in each image. When a normal camera is used, annotation information may be assigned to the image taken by the camera, but when a computationally captured image of the multi-pinhole camera 301 is acquired, it is difficult to assign annotation information because a person cannot know what is in the image even if he or she looks at the image. In addition, even if the learning process is performed using an image taken by a normal camera that is significantly different from the multi-pinhole camera 301, the identification unit 106 does not have high identification accuracy. Therefore, a database in which annotation information is added in advance to images captured by a normal camera is held as the learning database 102, and only the captured images are transformed in accordance with the PSF information of the multi-pinhole camera 301 to create a learning data set suited to the multi-pinhole camera 301, and the classification accuracy is improved by performing a learning process. To this end, the database correction unit 104 calculates the following corrected image y for an image z captured by a normal camera prepared in advance, using the PSF information acquired by the PSF information acquisition unit 103.

[0082] y=k*z

[0083] Here, k indicates the PSF information acquired by the PSF information acquisition unit 103, and * indicates a convolution operator.

[0084] The learning unit 105 performs a learning process using the corrected image calculated by the database correction unit 104 and the annotation information acquired from the learning database 102 (step S204). For example, when the identification unit 106 is constructed by a multi-layered neural network, the corrected image and the annotation information are used as teacher data to perform machine learning by deep learning. As a correction algorithm for prediction errors, a back propagation method or the like may be used. In this way, the learning unit 105 creates an image identification model for the identification unit 106 to identify images captured by the multi-pinhole camera 301. Since the corrected image is an image that matches the PSF information of the multi-pinhole camera 301, this learning enables learning suited to the multi-pinhole camera 301, and the identification unit 106 can perform highly accurate identification processing.

[0085] According to this embodiment, when the subject 302 is located at a distance less than a predetermined distance D from the multi-pinhole camera 301, the overlapping degree of the two subject images 302a, 302b captured through the first pinhole 301aa and the second pinhole 301ab falls within a predetermined range, so that the multiple subject images are visually incapable of being individually recognized by humans due to the overlapping of the multiple images, thereby protecting the privacy of the subject 302. Also, when the subject 302 is located at a distance equal to or greater than the predetermined distance D from the multi-pinhole camera 301, the overlapping degree of the two subject images 302b, 302c captured through the second pinhole 301ab and the third pinhole 301ac falls within a predetermined range, so that the multiple subject images are visually incapable of being individually recognized by humans due to the overlapping of the multiple images, thereby protecting the privacy of the subject 302. As a result, it is possible to protect the privacy of the subject 302 regardless of the distance between the multi-pinhole camera 301 and the subject 302.

[0086] (First Modification) In the above embodiment, the opening areas of the pinholes 301aa to 301ac are set to be equal to one another. However, the opening areas of the pinholes 301aa to 301ac may be made different from one another by making the diameters or shapes different.

[0087] Fig. 9 is a schematic diagram showing the structure of a multi-pinhole camera 301 according to a first modified example. In the example shown in Fig. 9, the aperture area of ​​the central pinhole 301ab is the smallest, the aperture area of ​​the right pinhole 301ac close to the pinhole 301ab is the next smallest, and the aperture area of ​​the left pinhole 301aa farthest from the pinhole 301ab is the largest. Since the larger the aperture area of ​​the pinhole, the greater the degree of blurring of the subject image, in this example, the greater the degree of blurring of the subject image 302a corresponding to the pinhole 301aa.

[0088] According to this modification, the aperture area of ​​each of the pinholes 301aa-301ac is different, and therefore the degree of blur of each of the subject images 302a-302c is also different. The captured image becomes more complicated due to the mixture of the subject images 302a-302c with different degrees of blur. As a result, it becomes more difficult for a person to visually identify the subject 302 included in the captured image, and therefore it is possible to further enhance the effect of protecting the privacy of the subject 302.

[0089] (Second Modification) FIG. 10 is a schematic diagram showing the structure of a multi-pinhole camera 301 according to a second modified example. In the example shown in FIG. 10, the main surface of the multi-pinhole mask 301a facing the light receiving surface of the image sensor 301b is divided into a total of four regions, the upper right region UR, the lower right region LR, the upper left region UL, and the lower left region LL. One pinhole 301ac is formed in the center of the main surface. Also, pinholes 301ab and 301aa are formed in each of the multiple regions UR, LR, UL, and LL. As a result, in the mask pattern of the multi-pinhole mask 301a, each of the multiple regions UR, LR, UL, and LL includes a pinhole set having a wide interval L1 including the first pinhole 301aa and the second pinhole 301ab, and a pinhole set having a narrow interval L2 including the third pinhole 301ac and the second pinhole 301ab. The number of regions into which the main surface of the multi-pinhole mask 301a is divided is not limited to four, but may be two or more.

[0090] In the multi-pinhole camera 301 in this figure, it is assumed that the image sensor 301b is located at the back and the subject 302 is located at the front. In this case, if the subject 302 is located in the lower right area, the light passing through the pinholes 301ab and 301aa in the area UL will not be received by the image sensor 301b if the image sensor 301b is not large enough or the angle of incidence is too large. In other words, if the pinholes of the multi-pinhole mask 301a are located only in the area UL, the subject 302 located in the lower right area will not be photographed, or only the light passing through the pinhole 301ac will be received, and no overlapping images will be formed, and the privacy of the subject 302 will not be protected. Similarly, if the subject 302 is located in the upper left area, the light passing through the pinholes 301ab and 301aa in the area LR will not be received by the image sensor 301b if the image sensor 301b is not large enough or the angle of incidence is too large. That is, if the pinholes of the multi-pinhole mask 301a exist only in the region LR, the subject 302 existing in the upper left region will not be photographed, or only the light passing through the pinholes 301ac will be received, and no overlapping image will be formed, and the privacy of the subject 302 cannot be protected. On the other hand, the multi-pinhole camera 301 of this modification includes a pinhole set having a wide interval L1 including the first pinhole 301aa and the second pinhole 301ab, and a pinhole set having a narrow interval L2 including the third pinhole 301ac and the second pinhole 301ab, in each of the multiple regions UR, LR, UL, and LL, so that the privacy of the subject can be protected regardless of the position of the subject 302.

[0091] According to this modification, even if the angle between the front direction of the multi-pinhole camera 301 and the position of the subject 302 is large (i.e., the subject is positioned at an angle relative to the front of the camera), the light from the subject 302 can reach the image sensor 301b via the pinhole set closer to the subject 302. As a result, it is possible to widen the angle of view of the multi-pinhole camera 301, which can realize the superimposition of multiple images.

[0092] (Third Modification) FIG. 11 is a schematic diagram showing the structure of a multi-pinhole camera 301 according to a first example of the third modified example. One pinhole 301ab is formed in the center of the main surface of the multi-pinhole mask 301a. In addition, three pinholes 301aa and one pinhole 301ac are formed around the pinhole 301ab. As a result, the number of pinhole sets (three sets in this example) with a wide interval L1 consisting of the first pinhole 301aa and the second pinhole 301ab is greater than the number of pinhole sets (one set in this example) with a narrow interval L2 consisting of the third pinhole 301ac and the second pinhole 301ab. The number of each pinhole set is not limited to this example.

[0093] According to this configuration, when the subject 302 is located at a distance less than the predetermined distance D from the multi-pinhole camera 301, it is possible to increase the number of effective overlapping images with a large amount of displacement of the subject image. As a result, it becomes more difficult for a human to visually identify the subject 302 included in the captured image, so it is possible to further enhance the effect of protecting the privacy of the subject 302. This is particularly effective for indoor operation where the distance between the multi-pinhole camera 301 and the subject 302 is likely to be close.

[0094] FIG. 12 is a schematic diagram showing the structure of a multi-pinhole camera 301 according to a second example of the third modified example. One pinhole 301ab is formed in the center of the main surface of the multi-pinhole mask 301a. In addition, one pinhole 301aa and three pinholes 301ac are formed around the pinhole 301ab. As a result, the number of pinhole sets (three sets in this example) with a narrow interval L2 consisting of the third pinhole 301ac and the second pinhole 301ab is greater than the number of pinhole sets (one set in this example) with a wide interval L1 consisting of the first pinhole 301aa and the second pinhole 301ab. The number of each pinhole set is not limited to this example.

[0095] According to this configuration, when the subject 302 is located at a distance equal to or greater than the predetermined distance D from the multi-pinhole camera 301, it is possible to increase the number of effective overlapping images with a small amount of deviation of the subject image. As a result, it becomes more difficult for a human to visually identify the subject 302 included in the captured image, so it is possible to further enhance the effect of protecting the privacy of the subject 302. This is particularly effective for outdoor operation where the distance between the multi-pinhole camera 301 and the subject 302 is likely to be large. [Industrial Applicability]

[0096] The learning method and classification method according to the present disclosure are particularly useful for image classification systems in environments where privacy protection of subjects is required.

Claims

1. A multi-pinhole camera, An image sensor; a mask arranged between a light receiving surface of the image sensor and a subject, the mask having a mask pattern in which a plurality of pinholes including a first pinhole, a second pinhole adjacent to the first pinhole, and a third pinhole adjacent to the second pinhole are formed; Equipped with a distance between the first pinhole and the second pinhole is set to a first distance; the first interval is an interval such that, when an image of a subject located at a distance less than a predetermined distance from the multi-pinhole camera is captured by the multi-pinhole camera, an overlapping degree of two subject images captured through the first pinhole and the second pinhole falls within a predetermined range of 30% or more and less than 90%, a distance between the second pinhole and the third pinhole is set to a second distance that is narrower than the first distance, A multi-pinhole camera, wherein the second interval is an interval such that when the subject located at a distance greater than or equal to the specified distance from the multi-pinhole camera is imaged by the multi-pinhole camera, the degree of overlap of two subject images imaged through the second pinhole and the third pinhole falls within the specified range.

2. when the subject located at a distance equal to or greater than the predetermined distance from the multi-pinhole camera is imaged by the multi-pinhole camera, a degree of overlap between the two subject images imaged through the first pinhole and the second pinhole falls outside the predetermined range, The multi-pinhole camera of claim 1, wherein when the subject located at a distance less than the specified distance from the multi-pinhole camera is imaged by the multi-pinhole camera, the degree of overlap of the two subject images imaged through the second pinhole and the third pinhole falls outside the specified range.

3. 3. The multi-pinhole camera according to claim 1, wherein the multi-pinhole camera does not have an optical system for forming an image of light from a subject on the image sensor.

4. 4. The multi-pinhole camera according to claim 1, wherein the first pinhole, the second pinhole, and the third pinhole have opening areas different from each other.

5. A multi-pinhole camera as described in any one of claims 1 to 4, wherein, when the mask is divided into a plurality of regions, in the mask pattern, each of the plurality of regions includes a pinhole set having the first pinhole and the second pinhole, and a pinhole set having the third pinhole and the second pinhole.

6. A multi-pinhole camera according to any one of claims 1 to 4, wherein in the mask pattern, the number of pinhole sets having the first pinhole and the second pinhole is greater than the number of pinhole sets having the third pinhole and the second pinhole.

7. A multi-pinhole camera according to any one of claims 1 to 4, wherein in the mask pattern, the number of pinhole sets having the third pinhole and the second pinhole is greater than the number of pinhole sets having the first pinhole and the second pinhole.

8. A multi-pinhole camera according to any one of claims 1 to 7; A recognition unit that recognizes an image captured by the multi-pinhole camera based on an image recognition model; an output unit that outputs a classification result by the classification unit; An image recognition system comprising:

Images