Multi-pinhole camera and image recognition system

Abstract

The interval between the pinhole (301aa) and the pinhole (301ab) is set to a first interval (L1) in which the degree of overlap of the subject images (302a, 302b) taken via the pinholes (301aa, 301ab) is within a prescribed range when a subject (302) located at a position at a distance of less than a prescribed distance (D) from the multi-pinhole camera (301) is imaged. The interval between the pinhole (301ab) and the pinhole (301ac) is set to a second interval (L2) narrower than the first interval (L1) in which the degree of overlap of the subject images (302b, 302c) taken via the pinholes (301ab, 301ac) is within a prescribed range when a subject (302) located at a position at a distance of greater than or equal to the prescribed distance (D) from the multi-pinhole camera (301) is imaged.

Description

Technical Field

[0001] This invention relates to an image recognition system, particularly in environments where privacy needs to be protected, such as homes or indoors, and a multi-pinhole camera used in the image recognition system. Background Technology

[0002] Patent Document 1 discloses an image recognition system in which a computationally captured image is input into a recognizer, which uses a learned recognition model to identify objects contained in the computationally captured image. Furthermore, Patent Document 1 also discloses, as an example of a camera for capturing computationally captured images, a light field camera using a multi-pinhole mask with multiple pinholes.

[0003] Computational camera images are intentionally blurred images that are difficult for humans to visually recognize due to factors such as the overlapping of multiple images from different viewpoints or the inability to focus the subject image without using a lens. Therefore, computational camera images are preferred for building image recognition systems, especially in environments where privacy needs to be protected, such as homes or indoors.

[0004] However, Patent Document 1 does not discuss in detail how to set the spacing between multiple pinholes in a multi-pinhole mask. Therefore, depending on the distance between the camera and the subject, it is possible that the subject's privacy cannot be protected due to the undesirable blurring.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication No. 2019 / 054092 Summary of the Invention

[0008] The purpose of this invention is to provide a technology that can protect the privacy of the subject in an image recognition system regardless of the distance between the camera and the subject.

[0009] An embodiment of the present invention relates to a multi-pinhole camera, comprising: an image sensor; and a mask disposed between the light-receiving surface of the image sensor and a subject, having a mask pattern having a plurality of pinholes including a first pinhole, a second pinhole adjacent to the first pinhole, and a third pinhole adjacent to the second pinhole. The interval between the first pinhole and the second pinhole is set as a first interval, which is an interval such that when the multi-pinhole camera is used to photograph a subject located at a distance less than a predetermined distance from the multi-pinhole camera, the overlap of the two subject images photographed via the first pinhole and the second pinhole is within a predetermined range. The interval between the second pinhole and the third pinhole is set as a second interval narrower than the first interval, which is an interval such that when the multi-pinhole camera is used to photograph the subject located at a distance greater than the predetermined distance from the multi-pinhole camera, the overlap of the two subject images photographed via the second pinhole and the third pinhole is within the predetermined range. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating the structure of the image recognition system involved in the implementation method.

[0011] Figure 2 This is a flowchart illustrating the main processing steps of an image recognition system.

[0012] Figure 3 This is a schematic diagram illustrating the construction of a multi-pinhole camera made without lenses.

[0013] Figure 4A This is a schematic diagram illustrating the structure of a multi-pinhole mask.

[0014] Figure 4B This is an example of a video image captured by a multi-hole camera.

[0015] Figure 4C This is an example of a video image captured by a multi-hole camera.

[0016] Figure 5A This is a schematic diagram illustrating the structure of a multi-pinhole mask.

[0017] Figure 5B This is an example of a video image captured by a multi-hole camera.

[0018] Figure 5C This is an example of a video image captured by a multi-hole camera.

[0019] Figure 6A This is a schematic diagram illustrating the structure of a multi-pinhole mask.

[0020] Figure 6BThis is an example of a video image captured by a multi-hole camera.

[0021] Figure 6C This is an example of a video image captured by a multi-hole camera.

[0022] Figure 7 It is a diagram showing the specified distance between a multi-hole camera and the subject.

[0023] Figure 8 This is a flowchart illustrating the main processing steps of the learning device.

[0024] Figure 9 This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the first modified example.

[0025] Figure 10 This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the second variation.

[0026] Figure 11 This is a schematic diagram illustrating the construction of the multi-pinhole camera involved in the first example of the third variation.

[0027] Figure 12 This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the second example of the third variation.

[0028] Figure 13A This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0029] Figure 13B This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0030] Figure 13C This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0031] Figure 13D This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0032] Figure 14A This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0033] Figure 14B This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0034] Figure 14C This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0035] Figure 14D This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0036] Figure 15This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0037] Figure 16 This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0038] Figure 17 This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0039] Figure 18A This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0040] Figure 18B This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0041] Figure 18C This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0042] Figure 18D This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0043] Figure 18E This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0044] Figure 18F This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0045] Figure 19A This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0046] Figure 19B This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example.

[0047] Figure 19C This is a schematic diagram illustrating the structure of the multi-pinhole camera involved in the modified example. Detailed Implementation

[0048] Basic knowledge of this invention

[0049] In environments such as homes and indoors, various recognition technologies are crucial, including human action recognition and operator identification. In recent years, deep learning has garnered significant attention for object recognition. Deep learning is machine learning that uses multi-layered neural networks, leveraging large amounts of learning data to achieve higher accuracy compared to previous methods. Image information is particularly effective in such object recognition. Various methods have been proposed to significantly improve object recognition capabilities by using cameras as input devices and performing deep learning with image information as input.

[0050] However, installing cameras in homes or other locations presents a problem: privacy is compromised if images are leaked to the outside world due to hacking or other means. Therefore, measures are needed to protect the privacy of the subjects even if images are leaked.

[0051] Computational imaging, captured using multi-pinhole cameras, is an intentionally blurred image that is difficult for humans to visually recognize due to factors such as the overlapping of multiple images from different viewpoints or the inability to focus the subject image without a lens. Therefore, computational imaging is preferred for building image recognition systems, especially in environments where privacy needs to be protected, such as homes or indoors.

[0052] In the image recognition system disclosed in Patent Document 1, a light field camera or similar device with multiple pinhole masks is used to photograph a target area, and the resulting computational image is input into a recognizer. The recognizer then uses a learned recognition model to identify objects contained in the computational image. By using a light field camera or similar device to photograph the target area, even if the photographed image is leaked to the outside world, the privacy of the subject can be protected because it is difficult for humans to visually recognize the computational image.

[0053] However, Patent Document 1 does not provide a detailed study on how to set the spacing between multiple pinholes in a multi-pinhole mask. Therefore, depending on the distance between the camera and the subject, the subject's privacy may not be protected due to the difficulty in achieving the desired visual recognition. For example, if the spacing between the pinholes is too wide, the offset relative to a small subject image will be too large when the distance between the camera and the subject is relatively far, resulting in multiple subject images not overlapping and thus failing to identify the subject's face. On the other hand, if the spacing between the pinholes is too narrow, the offset relative to a large subject image will be too small when the distance between the camera and the subject is relatively close, resulting in the identification of the subject's face.

[0054] In order to solve the aforementioned problem, the inventors of the present invention came to the following insight and conceived of the present invention: using a mask pattern with at least three pinholes spaced at different intervals, the intervals are set in such a way that the overlap of the subject images is within a specified range, thereby protecting the privacy of the subject regardless of the distance between the camera and the subject.

[0055] Next, various embodiments of the present invention will be described.

[0056] An embodiment of the present invention relates to a multi-pinhole camera, comprising: an image sensor; and a mask disposed between the light-receiving surface of the image sensor and a subject, having a mask pattern having a plurality of pinholes including a first pinhole, a second pinhole adjacent to the first pinhole, and a third pinhole adjacent to the second pinhole. The interval between the first pinhole and the second pinhole is set as a first interval, which is an interval such that when the multi-pinhole camera is used to photograph a subject located at a distance less than a predetermined distance from the multi-pinhole camera, the overlap of the two subject images photographed via the first pinhole and the second pinhole is within a predetermined range. The interval between the second pinhole and the third pinhole is set as a second interval narrower than the first interval, which is an interval such that when the multi-pinhole camera is used to photograph the subject located at a distance greater than the predetermined distance from the multi-pinhole camera, the overlap of the two subject images photographed via the second pinhole and the third pinhole is within the predetermined range.

[0057] According to this configuration, when the subject is located at a distance less than a predetermined distance from the multi-hole camera, by ensuring that the overlap between the two subject images captured through the first and second pinholes is within a predetermined range, the overlapping of multiple images renders the multiple subject images visually indistinguishable, thus protecting the subject's privacy. Furthermore, when the subject is located at a distance greater than the predetermined distance from the multi-hole camera, by ensuring that the overlap between the two subject images captured through the second and third pinholes is within a predetermined range, the overlapping of multiple images renders the multiple subject images visually indistinguishable, thus protecting the subject's privacy. As a result, the subject's privacy can be protected regardless of the distance between the multi-hole camera and the subject.

[0058] In the above embodiments, when the multi-pinhole camera is used to photograph the subject located at a distance greater than or equal to the multi-pinhole camera, the overlap of the two subject images captured through the first pinhole and the second pinhole is outside the specified range. When the multi-pinhole camera is used to photograph the subject located at a distance less than or equal to the multi-pinhole camera, the overlap of the two subject images captured through the second pinhole and the third pinhole is outside the specified range.

[0059] According to this configuration, even if the privacy of the subject cannot be fully protected by either the first interval or the second interval alone, the privacy of the subject can be protected regardless of the distance between the multi-pinhole camera and the subject by having the first interval and the second interval coexist within the same multi-pinhole mask.

[0060] In the above embodiment, the multi-pinhole camera does not have an optical system that images light from the subject onto the image sensor.

[0061] According to this configuration, because the multi-hole camera lacks an optical system that images light from the subject onto the image sensor, it is possible to intentionally generate blur in the captured image. As a result, it becomes more difficult for a person to visually identify the subject contained in the captured image, thus further improving the protection of the subject's privacy.

[0062] In the above embodiments, the opening areas of the first pinhole, the second pinhole, and the third pinhole may be different from each other.

[0063] According to this configuration, since the opening areas of each pinhole are different, the degree of blurring of each subject's image also varies. By allowing multiple subject images with varying degrees of blur to coexist, the captured image becomes more complex. As a result, it becomes more difficult for a human to visually identify the subject contained within the captured image, thus further enhancing the protection of the subject's privacy.

[0064] In the above embodiment, when the mask is divided into multiple regions, each region of the multiple regions in the mask pattern includes a pinhole group having the first pinhole and the second pinhole, and a pinhole group having the third pinhole and the second pinhole.

[0065] According to this configuration, even when the angle between the frontal direction of the multi-pinhole camera and the position of the subject is large, light from the subject can still reach the image sensor through the pinhole group closest to the subject. As a result, the viewing angle that enables multiple images can be expanded.

[0066] In the above embodiment, the number of pinhole groups having the first pinhole and the second pinhole in the mask pattern may also be greater than the number of pinhole groups having the third pinhole and the second pinhole.

[0067] According to this configuration, when the subject is located at a distance less than a predetermined distance from the multi-hole camera, the number of effective overlapping multiple images can be increased. As a result, it becomes more difficult for a person to visually identify the subject contained in the captured image, thus further improving the protection of the subject's privacy. This is particularly effective in indoor applications where the distance between the multi-hole camera and the subject can easily become too close.

[0068] In the above embodiment, the number of pinhole groups having the third pinhole and the second pinhole in the mask pattern may also be greater than the number of pinhole groups having the first pinhole and the second pinhole.

[0069] According to this configuration, when the subject is located at a distance greater than a specified distance from the multi-hole camera, the number of effective overlapping multiple images can be increased. As a result, it becomes more difficult for a person to visually identify the subject contained in the captured image, thus further improving the protection of the subject's privacy. This is particularly effective in outdoor applications where the distance between the multi-hole camera and the subject can easily increase.

[0070] Another embodiment of the present invention relates to an image recognition system comprising: a multi-pinhole camera as described in the above embodiment; a recognition unit for recognizing images captured by the multi-pinhole camera based on an image recognition model; and an output unit for outputting the recognition result of the recognition unit.

[0071] According to this configuration, when the subject is located at a distance less than a predetermined distance from the multi-hole camera, by ensuring that the overlap between the two subject images captured through the first and second pinholes is within a predetermined range, the overlapping of multiple images renders the multiple subject images visually indistinguishable, thus protecting the subject's privacy. Furthermore, when the subject is located at a distance greater than the predetermined distance from the multi-hole camera, by ensuring that the overlap between the two subject images captured through the second and third pinholes is within a predetermined range, the overlapping of multiple images renders the multiple subject images visually indistinguishable, thus protecting the subject's privacy. As a result, the subject's privacy can be protected regardless of the distance between the multi-hole camera and the subject.

[0072] This invention can also be implemented as a computer program that causes a computer to execute the characteristic components included in the method, or as a device or system that operates based on the computer program. Furthermore, needless to say, the computer program can be distributed via a computer-readable, non-transitory recording medium such as a CD-ROM, or via a communication network such as the Internet.

[0073] Furthermore, the embodiments described below are all specific examples of the present invention. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are merely specific examples and are not intended to limit the present invention. Moreover, among the constituent elements in the following embodiments, those not described in the independent claims representing the superior concept are described as arbitrary constituent elements. Furthermore, the contents of all embodiments can be combined arbitrarily.

[0074] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, elements given the same reference numerals in different drawings are the same or corresponding elements.

[0075] Embodiments of the present invention

[0076] Figure 1 This is a schematic diagram illustrating the structure of an image recognition system 10 according to an embodiment of the present invention. The image recognition system 10 includes a learning device 20 and a recognition device 30. The recognition device 30 includes a multi-pinhole camera 301, a recognition unit 106, and an output unit 107. The recognition unit 106 includes a processor such as a CPU and a memory such as semiconductor memory. The output unit 107 includes a display device or a speaker, etc. Furthermore, the learning device 20 includes a learning database 102, a PSF information acquisition unit 103, a database correction unit 104, and a learning unit 105. The learning database 102 includes a storage unit such as an HDD, SSD, or semiconductor memory. The PSF information acquisition unit 103, the database correction unit 104, and the learning unit 105 include a processor such as a CPU.

[0077] Figure 2 This is a flowchart illustrating the main processing steps of the image recognition system 10. The flowchart shows the image recognition processing performed by the recognition device 30. First, the multi-pinhole camera 301 captures an image of the object area containing the subject to be recognized, and inputs the calculated image obtained from this capture into the recognition unit 106 (step S101). Next, the recognition unit 106 uses a learned image recognition model to recognize the calculated image (step S102). This image recognition model is created through learning by the learning device 20. Then, the output unit 107 outputs the recognition result from the recognition unit 106. Details of each step will be described later.

[0078] Unlike conventional cameras that capture images that are easily visually identifiable by humans, the multi-hole camera 301 captures images that are difficult for humans to visually identify due to multiple images, i.e., computational images. Computational images are images that humans cannot identify even when they see the image itself, but which can be generated into images that can be recognized by humans or by the recognition unit 106 through image processing of the captured computational images.

[0079] Figure 3 This is a schematic diagram illustrating the structure of a multi-pinhole camera 301 constructed without a lens. Figure 3The multi-pinhole camera 301 shown has a multi-pinhole mask 301a and an image sensor 301b such as a CMOS. The multi-pinhole mask 301a is arranged between the image sensor 301b and the subject 302 at a fixed distance from the light-receiving surface of the image sensor 301b. A cover glass for IR cut-off and anti-reflection may also be arranged between the multi-pinhole mask 301a and the image sensor 301b. The cover glass may be separated from or in contact with the image sensor 301b. The cover glass may be separated from or in contact with the multi-pinhole mask 301a. The image sensor 301b, the cover glass, and the multi-pinhole mask 301a are supported by a housing in which brackets or struts for holding them are formed inside.

[0080] The multi-pinhole mask 301a has a plurality of pinholes 301aa to 301ac arranged linearly in this order. The interval between the pinhole 301aa and the pinhole 301ab is L1, and the interval between the pinhole 301ab and the pinhole 301ac is L2 (<L1). In the example of this embodiment, the shapes and opening areas of the pinholes 301aa to 301ac are equal to each other. The plurality of pinholes 301aa to 301ac are also referred to as multi-pinholes. The image sensor 301b acquires an image of the subject 302 through each of the pinholes 301aa to 301ac. The image acquired through a pinhole is called a pinhole image. In addition, the plurality of pinholes 301aa to 301ac do not necessarily need to be arranged linearly. For example, they may be in a form where each of the pinholes 301aa to 301ac is arranged at each vertex of a scalene triangle.

[0081] The pinhole images of the subject 302 are different according to the positions and sizes of the respective pinholes 301aa to 301ac. Therefore, the image sensor 301b acquires an overlapping image in a state where a plurality of pinhole images overlap slightly offset (i.e., a multiple image). The positional relationship of the plurality of pinholes 301aa to 301ac affects the positional relationship of the plurality of pinhole images projected onto the image sensor 301b (i.e., the overlapping degree of the multiple images), and the sizes of the pinholes 301aa to 301ac affect the blurring degree of the pinhole images.

[0082] By using the multi-pinhole mask 301a, multiple pinhole images with different positions and degrees of blur can be acquired by overlapping. In other words, it is possible to acquire computational imaging images that are difficult for humans to visually recognize due to the overlapping of multiple images from different viewpoints or the difficulty in focusing the subject image without a lens. Therefore, the captured image becomes a multi-image and blurred image, thus protecting the privacy of the subject 302 through these effects. Furthermore, by changing the number, position, and size of each pinhole, images with different degrees of overlap and blur can be acquired. That is, the structure can be configured so that the user can easily attach and detach the multi-pinhole mask 301a, multiple types of multi-pinhole masks 301a with different mask patterns can be prepared in advance, and the structure can be configured so that the user can freely replace the multi-pinhole mask 301a to be used.

[0083] In addition to replacing the mask, such changes to the mask can also be achieved through various other methods, including:

[0084] • Users can freely rotate the mask mounted in front of the image sensor;

[0085] • Users can make openings at any point on the board installed in front of the image sensor;

[0086] • By using liquid crystal masks that utilize spatial light modulators, the transmittance at each position within the mask can be arbitrarily set.

[0087] • A mask is formed using a stretchable material such as rubber. By applying external force, the mask is physically deformed, thereby changing the position and size of the holes. Examples of these deformations will be described below.

[0088]

[0089] Figures 13A to 13D This is a schematic diagram illustrating the structure of a multi-hole camera 301 whose mask can be rotated arbitrarily by the user. Figure 13A This represents an overview of the multi-hole camera 301 whose mask can be rotated arbitrarily by the user. Figure 13B A schematic diagram showing its cross-section is provided. The multi-hole camera 301 has a multi-hole mask 301a that can rotate relative to its housing 401, and a holding part 402 is connected to the multi-hole mask 301a. By holding and operating the holding part 402, the user can fix or rotate the multi-hole mask 301a relative to the housing 401. For such a mechanism, a screw can be provided at the holding part 402; tightening the screw fixes the multi-hole mask 301a, and loosening the screw rotates the multi-hole mask 301a. Figure 13C and Figure 13DThis diagram illustrates that the multi-pinhole mask 301a rotates 90 degrees when the gripping part 402 is rotated 90 degrees. Thus, by moving the gripping part 402, the multi-pinhole mask 301a can be rotated.

[0090] Additionally, in a multi-pinhole camera 301 where the user can arbitrarily rotate the mask, the multi-pinhole mask 301a can also be used as follows: Figure 13C The pinhole configuration is set up as shown, which is asymmetrical relative to rotation. In this way, a wide variety of multi-pinhole patterns can be achieved by rotating the mask by the user.

[0091] Of course, the structure of the multi-hole camera 301, which allows the user to rotate the mask at will, can also be a structure without the holding part 402. Figure 14A , Figure 14B This is a schematic diagram illustrating other structural examples of a multi-hole camera 301 whose mask can be rotated arbitrarily by the user. Figure 14A This provides an overview of other structural examples of the multi-hole camera 301, which allows the user to arbitrarily rotate the mask. Figure 14B A schematic diagram showing its cross-section is provided. A multi-pinhole mask 301a is fixed to the lens barrel 411. Additionally, an image sensor 301b is disposed on another lens barrel 412. The lens barrels 411 and 412 are rotatable via a threaded structure. Specifically, the lens barrel 412 exists outside the lens barrel 411, with a male thread etched on the outer side of its joint (the lens barrel 411) and a female thread etched on the inner side of the lens barrel 412. Furthermore, a fixing member 413 is first installed at the male thread of the lens barrel 411, followed by the lens barrel 412. The fixing member 413 also has a female thread etched on it, similar to that of the lens barrel 412. With this structure, when the lens barrel 411 is screwed into the lens barrel 412, the screwing depth varies depending on the position of the fixing member 413 within the lens barrel 411, thus allowing for a variable rotation angle of the multi-pinhole camera 301.

[0092] Figure 14C , Figure 14D This is a schematic diagram showing that the screwing depth of the fastener 413 changes depending on the position of the fastener 413 screwed into the lens barrel 411, and the rotation angle of the multi-pinhole camera 301 changes. Figure 14C This is a schematic diagram showing the fastener 413 being screwed deep into the lens barrel 411. Figure 14D This is a schematic diagram showing the case where the fastener 413 is only screwed in halfway into the lens barrel 411. (See diagram below.) Figure 14C As shown, when the fastener 413 is screwed deep into the lens barrel 411, the lens barrel 412 can be screwed deep into the lens barrel 411. On the other hand, as... Figure 14DAs shown, when the fastener 413 is screwed in only halfway into the lens barrel 411, the lens barrel 412 can only be screwed in halfway into the lens barrel 411. Therefore, the screwing depth of the fastener 413 varies depending on the position of the fastener 413 into the lens barrel 411, which can vary the rotation angle of the pinhole mask 301a.

[0093] <Example of user-defined mask opening>

[0094] Figure 15 This is a schematic diagram of the cross-section of a multi-pinhole camera 301 in which a hole is drilled at any location on a mask 301ab installed in front of the image sensor 301b. Figure 15 In this diagram, the same reference numerals are used for structural elements identical to those in Figure 13, and descriptions are omitted. Initially, there are no pinholes in mask 301ab. Users can create multiple holes at arbitrary locations using needles or similar tools on mask 301ab, thereby enabling the fabrication of multi-pinhole masks of arbitrary shapes.

[0095]

[0096] Figure 16 This is a schematic cross-section of a multi-pinhole camera 301, which uses a spatial light modulator 420 to arbitrarily set the transmittance at various positions within a mask. Figure 16 In China, regarding and Figure 15 The same structural elements are marked with the same reference numerals, and descriptions are omitted. The spatial light modulator 420 is composed of liquid crystal or the like and is capable of changing the transmittance of each pixel. This spatial light modulator 420 functions as a multi-pinhole mask. The change in transmittance can be controlled by the spatial light modulator control unit (not shown). Therefore, by allowing the user to select any pattern from a plurality of pre-prepared transmittance patterns, various mask patterns (multi-pinhole patterns) can be realized.

[0097] <Example of mask deformation by applying external force>

[0098] Figure 17 , Figures 18A to 18F This is a schematic diagram of the cross-section of a multi-hole camera 301, a structure that deforms a mask by applying external force. Figure 17 In China, regarding and Figure 15 The same structural elements are given the same reference symbols and the description is omitted. The multi-pinhole mask 301ac is composed of multiple masks 301a1, 301a2, and 301a3, each of which has an independent drive unit (not shown) for applying external force. Figures 18A to 18CThis is a schematic diagram illustrating the three masks 301a1, 301a2, and 301a3 constituting the multi-pinhole mask 301ac. Here, each mask is a combination of a fan shape and a ring shape. Of course, this structure is just one example; the shape is not limited to a fan shape, and the number of masks is not limited to three. One or more pinholes are formed in each mask. Alternatively, no pinholes may be formed in the mask. Mask 301a1 has two pinholes 301aa1 and 301aa2, mask 301a2 has one pinhole 301aa3, and mask 301a3 has two pinholes 301aa4 and 301aa5. By applying external force to move these three masks 301a1 to 301a3, various multi-pinhole patterns can be created.

[0099] Figures 18D to 18F Three types of multi-pinhole masks 301ac, each consisting of three masks 301a1 to 301a3, are shown. The driving units, omitted from the illustration, cause each mask 301a1 to 301a3 to move in different ways, thereby achieving… Figure 18D , Figure 18E The middle part forms a mask with 5 pinholes, in Figure 18F The mask 301ac is constructed with four pinholes. The drive unit for such a mask can be implemented using an ultrasonic motor or linear motor, which are widely used in autofocus and similar applications. This allows the number and position of the pinholes in the multi-pinhole mask 301ac to be changed by applying external force.

[0100] Of course, with multi-pinhole masks, it is also possible to change not only the number and location of the pinholes, but also their size. Figures 19A to 19C This is a schematic diagram illustrating the structure of a multi-pinhole mask 301ad in a multi-pinhole camera 301, used to explain the structure of a mask that deforms by applying external force. The multi-pinhole mask 301ad has multiple pinholes, is made of an elastic material, and has four drive units 421 to 424 at its four corners that can be independently controlled. Of course, the number of drive units does not need to be four. The position and size of the pinholes in the multi-pinhole mask 301ad can be changed by driving each drive unit 421 to 424.

[0101] Figure 19B This is a schematic diagram illustrating the case where drive units 421 to 424 are driven in the same direction. In this diagram, the arrows shown at drive units 421 to 424 indicate the driving direction of each drive unit. In this case, the multi-pinhole mask 301ad moves parallel to the driving direction of the drive units. On the other hand, Figure 19CThis is a schematic diagram illustrating the case where the drive units 421 to 424 are driven outward from the center of the multi-pinhole mask 301ad. In this case, the multi-pinhole mask 301ad is stretched elastically, thus increasing the size of the pinholes. Such drive units 421 to 424 can be implemented using ultrasonic motors or linear motors widely used in autofocus and the like. In this way, the position and size of the pinholes in the multi-pinhole mask 301ac can be changed by applying external force.

[0102] Furthermore, even with the same subject and the same multi-pinhole mask, the shape of the multiple images varies depending on the distance between the multi-pinhole camera 301 and the subject 302. Therefore, in the multi-pinhole camera 301 of this embodiment, by making the spacing L1 and L2 between the pinholes different, images with effective multiple image overlap can be captured regardless of the distance between the subject 302 and the multi-pinhole camera 301.

[0103] Explain the effectiveness of methods that make the spacing between pinholes L1 and L2 different.

[0104] Figure 4A This is a schematic diagram showing the structure of a multi-pinhole mask 301a when the spacing between the pinholes is unified into a narrow spacing L2. Figure 4B It is a diagram showing a photographed image taken when a subject 302 located at a distance from the multi-hole camera 301 is photographed. Figure 4C This diagram shows the captured images when a subject 302 located near the multi-hole camera 301 was photographed. The images captured through the pinholes 301aa to 301ac are respectively called subject images 302a to 302c.

[0105] In such Figure 4A When the spacing between the pinholes is uniformly set to a narrow interval L2 as shown, the offset of the subject images 302a to 302c captured through each pinhole 301aa to 301ac becomes smaller.

[0106] Therefore, when the subject 302 is located far from the multi-hole camera 301 and its image is small, the subject images captured through each pinhole effectively overlap with the subject images captured through adjacent pinholes. Thus, as... Figure 4B As shown, it is possible to capture images that protect the privacy of the subject 302 through effective superposition of multiple images.

[0107] On the other hand, when the subject 302 is located near the multi-hole camera 301 and the subject image is large, the subject images captured through each pinhole slightly overlap with the subject images captured through adjacent pinholes. However, the offset is too small relative to the size of the subject image, therefore... Figure 4CAs shown, the superposition of multiple images is not effective and cannot protect the privacy of the subject 302.

[0108] Figure 5A This is a schematic diagram showing the structure of a multi-pinhole mask 301a when the spacing between the pinholes is uniformly set to a wide spacing L1. Figure 5B It is a diagram showing a photographed image taken when a subject 302 located at a distance from the multi-hole camera 301 is photographed. Figure 5C It is a diagram showing a photographed image taken of a subject 302 located near the multi-hole camera 301.

[0109] exist Figure 5A When the spacing between the pinholes is uniformly set to a wide interval L1 as shown, the offset of the subject images 302a to 302c captured through each pinhole 301aa to 301ac becomes larger.

[0110] Therefore, when the subject 302 is located near the multi-hole camera 301 and the subject image is large, the subject images captured through each pinhole effectively overlap with the subject images captured through adjacent pinholes. Therefore, as... Figure 5C As shown, it is possible to capture images that protect the privacy of the subject 302 through effective superposition of multiple images.

[0111] On the other hand, when the subject 302 is located far away from the multi-pinhole camera 301 and the subject image is small, the offset is too large relative to the size of the subject image. Therefore, the subject images captured by each pinhole do not overlap with each other, and the privacy of the subject 302 cannot be protected.

[0112] Thus, by unifying the spacing between the pinholes to L1 or L2, it is difficult to capture images of all subjects 302 at different distances from the multi-pinhole camera 301 that protect privacy through effective superposition of multiple images.

[0113] The multi-pinhole camera 301 involved in this embodiment is characterized in that it includes a multi-pinhole mask 301a having a plurality of pinholes 301aa to 301ac, with the pinholes having different intervals L1 and L2.

[0114] Figure 6A This is a schematic diagram illustrating the structure of the multi-pinhole mask 301a according to this embodiment. The interval between pinholes 301aa and 301ab is L1, and the interval between pinholes 301ab and 301ac is L2 (<L1). Figure 6A The example shown has 3 pinholes 301aa to 301ac, but the number of pinholes is more than 3. Figure 6BIt is a diagram showing a photographed image taken when a subject 302 located at a distance from the multi-hole camera 301 is photographed. Figure 6C It is a diagram showing a photographed image taken of a subject 302 located near the multi-hole camera 301.

[0115] like Figure 6A As shown, the spacing L1 and L2 between the pinholes are different. Therefore, the subject images 302a to 302c captured through each pinhole 301aa to 301ac overlap with different offsets. Thus, even when the subject 302 is far from the multi-pinhole camera 301 and its image is small, and even when the subject 302 is near the multi-pinhole camera 301 and its image is large, the subject image captured through each pinhole overlaps with the subject image captured through the adjacent pinhole. As a result, as... Figure 6B and Figure 6C As shown, it is possible to capture images that protect the privacy of the subject 302 through effective superposition of multiple images.

[0116] Specifically, in cases where the subject 302 is located at a distance D or more from the multi-pinhole camera 301 and the subject image is small ( Figure 6B Under these conditions, the subject images 302b and 302c captured by the pinholes 301ab and 301ac with a narrow spacing L2 effectively overlap each other with a small offset. Additionally, in cases where the subject 302 is located near the multi-pinhole camera 301 at a distance less than a predetermined distance D, and the subject image is large... Figure 6C Under these conditions, the subject images 302a and 302b captured through pinholes 301aa and 30lab with a wide spacing L1 effectively overlap each other with a large offset. Thus, it is possible to capture an image that protects the privacy of the subject 302 through effective overlapping of multiple images. Furthermore, "effective overlap" means that the degree of overlap (area overlap) between the two subject images obtained through two adjacent pinholes is within a specified permissible range. For example, the lower limit of the permissible range is 30%, and the upper limit is 90%. That is, if the overlap between the two subject images obtained through two adjacent pinholes is more than 30% and less than 90%, a third party cannot individually identify the subject even if they see the captured image.

[0117] Figure 7This diagram illustrates the predetermined distance D between the multi-hole camera 301 and the subject 302. The predetermined distance D is set based on intervals L1 and L2, the distance between the image sensor 301b and the multi-hole mask 301a, and the sensor size and pixel pitch of the image sensor 301b, for example, set to 5m. In this example, when the subject 302 is located at a distance of less than 5m from the multi-hole camera 301, the wide interval L1 effectively functions, thereby effectively overlapping the subject images 302a and 302b. On the other hand, when the subject 302 is located at a distance of more than 5m from the multi-hole camera 301, the narrow interval L2 effectively functions, thereby effectively overlapping the subject images 302b and 302c.

[0118] Furthermore, in cases where a subject 302 located at a distance D or more from the multi-hole camera 301 is photographed using the multi-hole camera 301 ( Figure 6B Under these conditions, the overlap of the two subject images 302a and 302b captured by pinholes 301aa and 301ab with a wide interval L1 is outside the aforementioned specified range (less than the lower limit of 30%). Similarly, in the case where a subject 302 located at a distance less than a specified distance D from the multi-pinhole camera 301 is captured by the multi-pinhole camera 301 ( Figure 6C Under these conditions, the overlap of the two subject images 302b and 302c captured by pinholes 301ab and 301ac with a narrow spacing L2 is outside the aforementioned specified range (upper limit of 90% or more). Even in cases where the privacy of the subject 302 cannot be fully protected by only one of the spacings L1 and L2, by having the spacings L1 and L2 coexist within the same multi-pinhole mask 301a, the privacy of the subject 302 can be protected regardless of the distance between the multi-pinhole camera 301 and the subject 302.

[0119] In addition, such as Figure 3 As shown, it is desirable that the multi-pinhole camera 301 does not have an optical system (lens, prism, mirror, etc.) for imaging light from the subject 302 onto the image sensor 301b. By omitting the optical system, it is possible to achieve camera miniaturization and weight reduction, cost reduction, and improved design, and it is also possible to intentionally generate blur in the captured image. As a result, it is more difficult for a human to visually identify the subject 302 contained in the captured image, thus further improving the privacy protection of the subject 302.

[0120] Reference Figure 1The image recognition system 10 will now be described. The recognition unit 106 utilizes an image recognition model, which is the result of learning by the learning device 20, to identify the category information and positional information of subjects such as people (including human actions and expressions), cars, bicycles, or signals contained in an image of an object region captured by the multi-pinhole camera 301. In the learning process used to create the image recognition model, machine learning methods such as deep learning, which utilizes multi-layer neural networks, can be employed.

[0121] The output unit 107 outputs the result identified by the recognition unit 106. This can be configured to have an interface unit to present the recognition result to the user via images, text, or sound, or it can have a device control unit to change the control method based on the recognition result.

[0122] The learning device 20 includes a learning database 102, a PSF information acquisition unit 103, a database correction unit 104, and a learning unit 105. The learning device 20 performs learning to create an image recognition model for use by the recognition unit 106, corresponding to the PSF information of the computational camera 101 actually used in the shooting of the object area.

[0123] in addition, Figure 8 This is a flowchart illustrating the main processing steps of the learning device 20 in the image recognition system 10.

[0124] First, the PSF information acquisition unit 103 acquires PSF information, which is information representing the shape of the computational image captured by the computational camera 101 (step S201). In this regard, the multi-hole camera 301 can have a transmitting unit, and the PSF information acquisition unit 103 can have a receiving unit, exchanging PSF information via wired or wireless means. Alternatively, the PSF information acquisition unit 103 can have an interface, allowing the user to input PSF information into the PSF information acquisition unit 103.

[0125] The Point Spread Function (PSF) is the transfer function of cameras such as multi-hole cameras or coded aperture cameras, and it is represented by the following relationship.

[0126] y = k * x

[0127] Here, y is a computationally blurred image captured by a multi-hole camera 301, k is the PSF, and x is a normal image obtained by capturing the scene using a normal camera without blur. Additionally, * represents the convolution operator.

[0128] Regarding the PSF (Pulse Response Signal), it can be obtained by capturing images of the point light source using a multi-hole camera 301. This can be seen from the PSF corresponding to the camera's impulse response. In other words, the captured image of the point light source obtained by capturing images of the point light source using the multi-hole camera 301 is itself a PSF of the captured image information of the multi-hole camera 301. Here, as the captured image of the point light source, it is desirable to use a difference image that includes the captured image of the point light source in its lit state and the captured image of the point light source in its off state.

[0129] Next, the database correction unit 104 acquires the non-blurred normal images contained in the learning database 102, and the learning unit 105 acquires the annotation information contained in the learning database 102 (step S202).

[0130] Next, the database correction unit 104 uses the PSF information acquired by the PSF information acquisition unit 103 to correct the learning database 102 (step S203). For example, when the recognition unit 106 recognizes human actions in the environment, the learning database 102 holds multiple ordinary images taken with a normal camera that does not have blur, and annotation information (correct answer labels) assigned to each image, such as where the person is in each image and what kind of action they are taking. When using a normal camera, it is sufficient to assign annotation information to the images taken with that camera. However, when acquiring the computed image from the multi-hole camera 301, it is difficult to assign annotation information because the person does not know what was captured when they see the image. In addition, even if learning processing is performed using images taken with a normal camera that is very different from the multi-hole camera 301, the recognition accuracy of the recognition unit 106 will not improve. Therefore, a learning database 102 is maintained, containing pre-annotated information for images captured using a conventional camera. The images are distorted only in accordance with the PSF information of the multi-hole camera 301, thereby creating a learning dataset that matches the multi-hole camera 301 and performing learning processing to improve recognition accuracy. To this end, the database correction unit 104 calculates the following corrected image y for the pre-prepared captured image z obtained using a conventional camera, using the PSF information acquired by the PSF information acquisition unit 103.

[0131] y = k * z

[0132] Here, k represents the PSF information acquired by the PSF information acquisition unit 103, and * represents the convolution operator.

[0133] The learning unit 105 performs learning processing (step S204) using the corrected image calculated by the database correction unit 104 and the annotation information obtained from the learning database 102. For example, if the recognition unit 106 is constructed using a multi-layer neural network, the corrected image and annotation information are used as training data (teacher data) to perform deep learning-based machine learning. As a correction algorithm for prediction errors, methods such as back propagation can be used. Thus, the learning unit 105 creates an image recognition model for the recognition unit 106 to recognize images captured by the multi-hole camera 301. The corrected image becomes an image that matches the PSF information of the multi-hole camera 301, so through such learning, learning suitable for the multi-hole camera 301 can be performed, and the recognition unit 106 can perform high-precision recognition processing.

[0134] According to this embodiment, when the subject 302 is located at a distance D less than that of the multi-hole camera 301, the overlap of the two subject images 302a and 302b captured by the first pinhole 301aa and the second pinhole 301ab is within a specified range. Therefore, due to the overlap of multiple images, the multiple subject images become visually indistinguishable to a human, thus protecting the privacy of the subject 302. Furthermore, when the subject 302 is located at a distance D or more than that of the multi-hole camera 301, the overlap of the two subject images 302b and 302c captured by the second pinhole 301ab and the third pinhole 301ac is within a specified range. Again, due to the overlap of multiple images, the multiple subject images become visually indistinguishable to a human, thus protecting the privacy of the subject 302. As a result, the privacy of the subject 302 can be protected regardless of the distance between the multi-hole camera 301 and the subject 302.

[0135] First variation

[0136] In the above embodiment, the opening areas of pinholes 301aa to 301ac are set to be equal to each other, but the opening areas of pinholes 301aa to 301ac can also be made different by making their diameters or shapes different.

[0137] Figure 9 This is a schematic diagram illustrating the structure of the multi-pinhole camera 301 involved in the first modified example. Figure 9In the example shown, the opening area of ​​the central pinhole 301ab is set to the smallest, the opening area of ​​the pinhole 301ac to the right of pinhole 301ab is set to the second smallest, and the opening area of ​​the pinhole 301aa to the left of pinhole 301ab, which is farthest from pinhole 301ab, is set to the largest. The larger the opening area of ​​the pinhole, the greater the blur of the subject image. Therefore, according to this example, the subject image 302a corresponding to pinhole 301aa has the greatest blur.

[0138] According to this modified example, since the opening areas of each pinhole 301aa to 301ac are different, the degree of blurring of each subject image 302a to 302c is also different. Multiple subject images 302a to 302c with varying degrees of blurring coexist, thus making the captured image more complex. As a result, it becomes more difficult for a person to visually identify the subject 302 contained in the captured image, thereby further improving the privacy protection of the subject 302.

[0139] Second variation

[0140] Figure 10 This is a schematic diagram illustrating the structure of the multi-pinhole camera 301 involved in the second modification. Figure 10 In the example shown, the main surface of the multi-pinhole mask 301a facing the light-receiving surface of the image sensor 301b is divided into four regions: the upper right region UR, the lower right region LR, the upper left region UL, and the lower left region LL. A pinhole 301ac is formed in the center of this main surface. Additionally, pinholes 301ab and 301aa are formed in each of the multiple regions UR, LR, UL, and LL. Thus, in the mask pattern of the multi-pinhole mask 301a, each of the multiple regions UR, LR, UL, and LL includes a group of pinholes with a wide interval L1 between the first pinhole 301aa and the second pinhole 301ab, and a group of pinholes with a narrow interval L2 between the third pinhole 301ac and the second pinhole 301ab. Furthermore, the number of regions dividing the main surface of the multi-pinhole mask 301a is not limited to four; two or more are sufficient.

[0141] In the multi-pinhole camera 301 shown in the figure, an image sensor 301b is located on the inner side, and a subject 302 is located on the front side. When the subject 302 is located in the lower right region, light passing through pinholes 301ab and 301aa in region UL will not be received by the image sensor 301b if it is not large enough or if the angle of incidence is too large. In other words, when the pinholes of the multi-pinhole mask 301a only exist in region UL, the subject 302 in the lower right region will not be photographed, or only light passing through pinhole 301ac will be received, failing to create a multiple image and thus failing to protect the privacy of the subject 302. Similarly, when the subject 302 is located in the upper left region, light passing through pinholes 301ab and 301aa in region LR will not be received by the image sensor 301b if it is not large enough or if the angle of incidence is too large. In other words, when the pinholes of the multi-pinhole mask 301a exist only in region LR, the subject 302 in the upper left region is not photographed, or only light passing through pinhole 301ac is received, thus not forming a multiple image and failing to protect the privacy of the subject 302. On the other hand, the multi-pinhole camera 301 of this modified example includes a group of pinholes with a wide interval L1 having a first pinhole 301aa and a second pinhole 301ab, and a group of pinholes with a narrow interval L2 having a third pinhole 301ac and a second pinhole 301ab in each of the multiple regions UR, LR, UL, and LL, so that the privacy of the subject can be protected regardless of the position of the subject 302.

[0142] According to this modified example, even when the angle between the frontal direction of the multi-pinhole camera 301 and the position of the subject 302 is large (i.e., the subject is positioned at an angle relative to the front of the camera), light from the subject 302 can still reach the image sensor 301b through the pinhole group on the side closest to the subject 302. As a result, the viewing angle of the multi-pinhole camera 301, which is capable of achieving multiple image superposition, can be expanded.

[0143] Third variation

[0144] Figure 11 This is a schematic diagram illustrating the structure of the multi-pinhole camera 301 involved in the first example of the third modification. One pinhole 301ab is formed in the center of the main surface of the multi-pinhole mask 301a. Furthermore, three pinholes 301aa and one pinhole 301ac are formed around the pinhole 301ab. Thus, the number of pinhole groups with a wide interval L1 formed by the first pinhole 301aa and the second pinhole 301ab (three groups in this example) is greater than the number of pinhole groups with a narrow interval L2 formed by the third pinhole 301ac and the second pinhole 301ab (one group in this example). Furthermore, the number of each pinhole group is not limited in this example.

[0145] According to this structure, when the subject 302 is located at a distance D less than the specified distance between it and the multi-hole camera 301, the number of effective overlapping multiple images with a large offset of the subject image can be increased. As a result, it becomes more difficult for a person to visually identify the subject 302 contained in the captured image, thus further improving the privacy protection of the subject 302. This is particularly effective in indoor applications where the distance between the multi-hole camera 301 and the subject 302 can easily become close.

[0146] Figure 12 This is a schematic diagram illustrating the structure of the multi-pinhole camera 301 involved in the second example of the third modification. One pinhole 301ab is formed in the center of the main surface of the multi-pinhole mask 301a. Additionally, one pinhole 301aa and three pinholes 301ac are formed around pinhole 301ab. Thus, the number of pinhole groups with a narrow interval L2 formed by the third pinhole 301ac and the second pinhole 301ab (three groups in this example) is greater than the number of pinhole groups with a wide interval L1 formed by the first pinhole 301aa and the second pinhole 301ab (one group in this example). Furthermore, the number of each pinhole group is not limited in this example.

[0147] According to this structure, when the subject 302 is located at a distance D or more from the multi-hole camera 301, the number of effective overlapping multiple images with small subject image offsets can be increased. As a result, it becomes more difficult for a human to visually identify the subject 302 contained in the captured image, thus further improving the privacy protection of the subject 302. This is particularly effective in outdoor applications where the distance between the multi-hole camera 301 and the subject 302 can easily increase.

[0148] Industrial availability

[0149] The learning and recognition methods involved in this invention are particularly useful in image recognition systems in environments where the privacy of the subject needs to be protected.

Claims

1. A multi-pinhole camera, comprising: Image sensor; as well as A mask is disposed between the light-receiving surface of the image sensor and the subject, and has a mask pattern forming a plurality of pinholes including a first pinhole, a second pinhole adjacent to the first pinhole, and a third pinhole adjacent to the second pinhole. The distance between the first pinhole and the second pinhole is set as the first distance. The first interval is the interval between two subject images captured by the first and second pinhole cameras, where the overlap is within a specified range when the subject is photographed at a distance less than a specified distance from the multi-pinhole camera. The specified range is 30% or more but less than 90%. The distance between the second pinhole and the third pinhole is set to a second distance that is narrower than the first distance. The second interval is the interval within the specified range of overlap between the two subject images captured through the second pinhole and the third pinhole when the subject is photographed at a position located at a distance greater than or equal to the specified distance from the multi-pinhole camera using the multi-pinhole camera.

2. The multi-pinhole camera according to claim 1, characterized in that: When the multi-pinhole camera is used to photograph the subject located at a distance greater than the specified distance from the multi-pinhole camera, the overlap between the two subject images captured through the first pinhole and the second pinhole is outside the specified range. When the subject is photographed at a distance less than the specified distance from the multi-hole camera using the multi-hole camera, the overlap of the two subject images captured through the second and third pinholes is outside the specified range.

3. The multi-pinhole camera according to claim 1 or 2, characterized in that: The multi-pinhole camera does not have an optical system that images light from the subject onto the image sensor.

4. The multi-pinhole camera according to claim 1 or 2, characterized in that: The opening areas of the first pinhole, the second pinhole, and the third pinhole are all different.

5. The multi-pinhole camera according to claim 1 or 2, characterized in that: When the mask is divided into multiple regions, in the mask pattern, each region of the multiple regions includes a pinhole group having the first pinhole and the second pinhole, and a pinhole group having the third pinhole and the second pinhole.

6. The multi-pinhole camera according to claim 1 or 2, characterized in that: In the mask pattern, the number of pinhole groups having the first pinhole and the second pinhole is greater than the number of pinhole groups having the third pinhole and the second pinhole.

7. The multi-pinhole camera according to claim 1 or 2, characterized in that: In the mask pattern, the number of pinhole groups having the third pinhole and the second pinhole is greater than the number of pinhole groups having the first pinhole and the second pinhole.

8. An image recognition system, comprising: The multi-pinhole camera according to any one of claims 1 to 7; The recognition unit identifies images captured by the multi-pinhole camera based on an image recognition model; as well as The output unit outputs the recognition result of the recognition unit.

Images