Control device and control method

Abstract

A control device (2) is provided with: an image acquiring unit (21) for acquiring an image obtained by image capture performed by an image capture device in which a mask having a plurality of pin holes is disposed to cover a light receiving surface of an image capture element; a control unit (23) for controlling at least one of size and interval of two pin holes among the plurality of pin holes on the basis of the image; and an output unit (25) for outputting an image obtained by the image capture device after at least one of size and interval of the two pin holes is controlled.

Description

Technical Field

[0001] The present disclosure relates to a technique for controlling an imaging device in which a mask having a plurality of pinholes is arranged to cover a light receiving surface of an image sensor.

Background Art

[0002] For example, Patent Document 1 discloses a system that executes actions including a step of automatically discriminating a region of interest in the foreground of a scene obtained by a privacy camera, a step of determining one privacy mode from a plurality of privacy modes including background blurring and background replacement, and when the determined privacy mode is background blurring, an image is generated by setting a depth of field of the privacy camera so that an object of interest in the scene in the foreground appears in focus and an object in the background of the scene does not appear in focus, and a step of displaying the image.

[0003] However, in the above conventional technique, when an object of interest in the foreground moves to the background, an object in the background that is the target of privacy protection also appears to be in focus, so it is difficult to protect the privacy of the subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide a technique capable of protecting the privacy of a subject even when the distance from the imaging device to the subject changes.

[0006] A control device according to one aspect of the present disclosure includes: an acquisition unit that acquires an image obtained by imaging of an imaging device in which a mask having a plurality of pinholes is arranged to cover the light-receiving surface of an image sensor; a control unit that controls at least one of the size and spacing of two of the plurality of pinholes based on the image; and an output unit that outputs an image obtained by the imaging device after at least one of the size and spacing of the two pinholes has been controlled.

[0007] According to this disclosure, the privacy of the subject can be protected even when the distance from the imaging device to the subject changes. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the overall configuration of the control system in Embodiment 1 of the present disclosure. [Figure 2] This diagram schematically shows the structure of a multi-pinhole camera, which is an example of an imaging device. [Figure 3] This figure shows the image of the subject that is formed on the light-receiving surface of the image sensor when the distance between the multi-pinhole mask and the subject is relatively large. [Figure 4] This figure shows the image of the subject that is formed on the light-receiving surface of the image sensor when the distance between the multi-pinhole mask and the subject is relatively short. [Figure 5] This is a schematic diagram illustrating the standard pinhole width and standard distance. [Figure 6] This is a schematic diagram illustrating the method for calculating the pinhole width in this embodiment 1. [Figure 7] This is a flowchart illustrating the control process in the control device according to Embodiment 1 of this disclosure. [Figure 8] This is a block diagram showing an example of the overall configuration of the control system in Embodiment 2 of the present disclosure. [Figure 9] This is a flowchart illustrating the control process in the control device according to Embodiment 2 of this disclosure. [Figure 10] This is a block diagram showing an example of the overall configuration of the control system in Embodiment 3 of the present disclosure. [Figure 11] This is a flowchart illustrating the control process in the control device according to Embodiment 3 of this disclosure. [Modes for carrying out the invention]

[0009] (Knowledge that forms the basis of this disclosure) In homes and indoors, various recognition technologies are important, such as recognizing people's actions or recognizing the person operating a device. In recent years, a technology called deep learning has attracted attention for object recognition. Deep learning is a machine learning method that uses a multi-layered neural network, and by utilizing a large amount of training data, it is possible to achieve higher accuracy in recognition performance compared to conventional methods. Image information is particularly effective in such object recognition. Various methods have been proposed that significantly improve conventional object recognition capabilities by using a camera as an input device and performing deep learning with image information as input.

[0010] However, placing cameras inside homes and other public spaces presents the challenge of privacy violations if captured images are leaked externally due to hacking or other means. Therefore, measures are needed to protect the privacy of the subjects even if captured images are leaked externally.

[0011] Furthermore, while cameras are placed outdoors or inside stores for purposes such as collecting big data, collecting images that show faces or other features that can identify individuals is undesirable from the standpoint of protecting privacy.

[0012] As described above, with conventional technology, when an object of interest in the foreground moves to the background, it appears as if the background object, which is subject to privacy protection, is also in focus, making it difficult to adequately protect the subject's privacy. Therefore, conventional technology requires measures to protect the subject's privacy when the distance from the imaging device to the subject changes. Here, "subject" refers to an object whose privacy is to be protected.

[0013] For example, a multi-pinhole camera is used to obtain blurred images that are difficult for humans to visually recognize. A multi-pinhole camera intentionally creates a blurred image by superimposing multiple images from different viewpoints. The multi-pinhole camera is described in the paper "Cognitive Sensing: Depth Estimation by Deep Learning from Multi-Pinhole Camera Images" (Satoshi Sato et al., Panasonic Technical Review, May 15, 2018, Vol. 64, No. 1, pp. 33-38).

[0014] In a multi-pinhole camera, if the distance between the camera and the subject is relatively far, multiple subject images will be superimposed, resulting in an image that is difficult for humans to visually recognize. On the other hand, if the distance between the multi-pinhole camera and the subject is relatively close, the amount of misalignment between the multiple subject images will become too large, reducing the superimposed area and potentially resulting in an image that humans can visually recognize.

[0015] Thus, depending on the distance between the multi-pinhole camera and the subject, images obtained by a multi-pinhole camera may not adequately protect the subject's privacy.

[0016] In order to solve the above problems, a control device according to an aspect of the present disclosure includes an acquisition unit that acquires an image obtained by imaging an imaging device in which a mask having a plurality of pinholes is arranged to cover a light receiving surface of an image sensor, a control unit that controls at least one of the size and the interval of two of the plurality of pinholes based on the image, and an output unit that outputs an image obtained by the imaging device after at least one of the size and the interval of the two pinholes is controlled.

[0017] According to this configuration, at least one of the size and the interval of two of the plurality of pinholes is controlled based on an image obtained by imaging an imaging device in which a mask having a plurality of pinholes is arranged to cover a light receiving surface of an image sensor. Therefore, by controlling at least one of the size and the interval of two of the plurality of pinholes based on the image, an image in which it is difficult for a human to visually recognize a subject can be obtained, so that the privacy of the subject can be protected even when the distance from the imaging device to the subject changes.

[0018] Further, in the above control device, the control unit may calculate the distance between the subject reflected in the image and the imaging device, and control at least one of the size and the interval of the two pinholes based on the distance.

[0019] As the subject approaches the imaging device, the degree of overlap of the subject images on the light receiving surface of the image sensor decreases, and an image in which a human can visually recognize the subject can be obtained. However, according to this configuration, since at least one of the size and the interval of the two pinholes is controlled based on the distance between the subject reflected in the image and the imaging device, an image in which it is difficult for a human to visually recognize the subject can be obtained even when the distance between the subject and the imaging device changes.

[0020] Furthermore, in the control device described above, the control unit may calculate the distance by inputting the acquired image into an estimation model that has been trained using machine learning to take the image in which the subject is captured as input and output the distance between the subject and the imaging device.

[0021] With this configuration, the distance is calculated by inputting the acquired image into an estimation model that has been trained using machine learning to take an image of the subject as input and output the distance between the subject and the imaging device. Therefore, the distance between the subject and the imaging device can be easily calculated from the image.

[0022] Furthermore, in the control device described above, the control unit may, when controlling the size, increase the size as the distance decreases, and when controlling the interval, narrow the interval as the distance decreases.

[0023] In this configuration, if the sizes of the two pinholes are controlled, the size of the two pinholes increases as the distance between the subject and the imaging device decreases. If the spacing between the two pinholes is controlled, the spacing between the two pinholes decreases as the distance between the subject and the imaging device decreases. Therefore, even if the subject approaches the imaging device, it is possible to obtain an image that is difficult for a human to visually recognize as the subject.

[0024] Furthermore, in the control device described above, the control unit may calculate the interval between the two pinholes to be controlled based on the image, and if the calculated interval is shorter than a threshold, it may increase the size, and if the calculated interval is greater than or equal to the threshold, it may narrow the interval.

[0025] As the distance between the two pinholes narrows, the degree of overlap between the two subject images on the image sensor's light-receiving surface increases, potentially causing the two subject images to effectively merge into one, making it possible for humans to visually recognize the subject. However, with this configuration, if the calculated interval is shorter than the threshold, the size of the two pinholes is increased rather than the distance between them being controlled. Therefore, even if the distance between the two pinholes narrows and the degree of overlap between the two subject images on the image sensor's light-receiving surface increases, it is still possible to obtain an image in which it is difficult for humans to visually recognize the subject.

[0026] Furthermore, in the control device described above, the control unit may calculate the degree of blurring of the subject in the image and control at least one of the size and spacing of the two pinholes based on the degree of blurring.

[0027] With this configuration, if the degree of blurring of the subject in the image is such that a human can visually recognize the subject, then by controlling at least one of the sizes and spacing of the two pinholes, it is possible to obtain an image in which a human has difficulty visually recognizing the subject.

[0028] Furthermore, in the control device described above, the control unit may, after controlling one of the sizes and spacings of the two pinholes, calculate the degree of blurring of the subject in the image, and if the degree of blurring is below a threshold, control the uncontrolled size and spacing of the two pinholes.

[0029] With this configuration, even if the degree of blurring of the subject in the image is below a threshold after one of the two pinhole sizes and spacings is controlled, the uncontrolled pinhole size and spacing is then controlled, increasing the degree of blurring of the subject in the image and making it difficult for humans to visually recognize the subject.

[0030] Furthermore, in the control device described above, the control unit may calculate the degree of blur by inputting the acquired image into an estimation model that has been trained using machine learning to take the image in which the subject is reflected as input and output the degree of blur of the subject reflected in the image.

[0031] With this configuration, the degree of blur is calculated by inputting the acquired image into an estimation model that has been trained using machine learning to take an image containing a subject as input and output the degree of blur of the subject in the image. Therefore, the degree of blur of a subject can be easily calculated from an image.

[0032] Furthermore, in the control device described above, if multiple subjects are visible in the image, the control unit may calculate the distance between each of the multiple subjects and the imaging device, and control at least one of the sizes and spacings of the two pinholes based on the calculated distance between the subject with the shortest distance and the imaging device.

[0033] With this configuration, even when multiple subjects are present in the image, at least one of the sizes and spacing of the two pinholes is controlled based on the distance between the subject closest to the imaging device and the imaging device itself. This increases the degree of blurring for all subjects in the image, resulting in an image that is difficult for humans to visually recognize.

[0034] Furthermore, in the control device described above, the image output by the output unit may be a new image obtained by a new image captured by the imaging device after the control unit has controlled at least one of the size and spacing of the two pinholes.

[0035] Furthermore, this disclosure can be implemented not only as a control device having the characteristic configuration described above, but also as a control method that performs characteristic processing corresponding to the characteristic configuration of a control system. It can also be implemented as a computer program that causes a computer to execute the characteristic processing included in such a control method. Therefore, the same effects as the control device described above can be achieved in the following other embodiments.

[0036] A control method according to another aspect of the present disclosure involves a computer acquiring an image obtained by imaging of an imaging device in which a mask having a plurality of pinholes is arranged to cover the light-receiving surface of an image sensor, controlling at least one of the size and spacing of two of the plurality of pinholes based on the image, and outputting an image obtained by the imaging device after at least one of the size and spacing of the two pinholes has been controlled.

[0037] The embodiments described below are all specific examples of this disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components. In addition, the contents of each embodiment can be combined.

[0038] (Embodiment 1) Figure 1 is a block diagram showing an example of the overall configuration of the control system 1 in Embodiment 1 of the present disclosure. The control system 1 comprises a control device 2 and an imaging device 3.

[0039] Unlike a normal camera that captures a normal image without blur, the imaging device 3 captures an image with blur. The image obtained by the imaging device 3 is an image in which the subject cannot be recognized by a human eye due to the intentionally created blur.

[0040] The imaging device 3 is, for example, a multi-pinhole camera in which a mask having a mask pattern with multiple pinholes is positioned to cover the light-receiving surface of the image sensor. In other words, the mask pattern is positioned between the subject and the light-receiving surface.

[0041] Figure 2 is a schematic diagram showing the structure of a multi-pinhole camera 30, which is an example of an imaging device 3. Figure 2 is a top view of the multi-pinhole camera 30 and the subject 4, which is a person.

[0042] The multi-pinhole camera 30 shown in Figure 2 comprises a multi-pinhole mask 31 and an image sensor 32 such as a CMOS. The multi-pinhole mask 31 is positioned at a certain distance from the light-receiving surface of the image sensor 32. The multi-pinhole mask 31 has a plurality of pinholes 33 arranged randomly or at equal intervals. The image sensor 32 acquires an image of the subject 4 by capturing it through each pinhole 33. The image acquired through the pinholes is called a pinhole image.

[0043] Since the pinhole image differs depending on the position and size of each pinhole 33, the image sensor 32 acquires a superimposed image in which multiple pinhole images are slightly shifted and overlapping (multiple images). The positional relationship of the multiple pinholes 33 affects the positional relationship of the multiple pinhole images projected onto the image sensor 32 (i.e., the degree of superposition of the multiple images), and the size of the pinholes 33 affects the degree of blurring of the pinhole images.

[0044] By using the multi-pinhole mask 31, it is possible to acquire multiple pinhole images with different positions and degrees of blur by superimposing them. In other words, it is possible to acquire captured images in which multiple images and blur are intentionally created. As a result, the captured image becomes a multiple-image and blurred image, and images in which the privacy of the subject is protected by this blur can be acquired.

[0045] The multi-pinhole mask 31 is a liquid crystal mask that utilizes a spatial light modulator or the like. By using a liquid crystal mask as the multi-pinhole mask 31, it is possible to arbitrarily set the transmittance at each position within the mask, and it is possible to change the position and size (aperture area) of the pinholes 33.

[0046] Furthermore, it is desirable that the multi-pinhole camera 30 does not have an optical system (lenses, prisms, mirrors, etc.) for focusing light from the subject onto the image sensor 32. By omitting the optical system, the multi-pinhole camera 30 can be made smaller, lighter, cost-reduced, and its design can be improved.

[0047] In Figure 2, two pinholes 33 are arranged horizontally, but this disclosure is not limited to this, and the multi-pinhole camera 30 may have three or more pinholes 33.

[0048] Next, we will explain the distance between the multi-pinhole mask 31 and the subject 4, and the degree of overlap of the subject image on the image sensor 32.

[0049] Figure 3 shows the image of the subject formed on the light-receiving surface of the image sensor 32 when the distance between the multi-pinhole mask 31 and the subject 4 is relatively far, and Figure 4 shows the image of the subject formed on the light-receiving surface of the image sensor 32 when the distance between the multi-pinhole mask 31 and the subject 4 is relatively close.

[0050] The multi-pinhole mask 31 includes a first pinhole 33A and a second pinhole 33B arranged horizontally. The first pinhole 33A forms a first subject image 4A on the light-receiving surface of the image sensor 32, and the second pinhole 33B forms a second subject image 4B on the light-receiving surface of the image sensor 32.

[0051] As shown in Figure 3, when the distance between the multi-pinhole mask 31 and the subject 4 is relatively large, the angle θ between the light incident on the first pinhole 33A and the light incident on the second pinhole 33B becomes small, and the degree of overlap between the first subject image 4A and the second subject image 4B increases. As a result, the image captured by the image sensor 32 becomes an indistinct image in which two subject images that are misaligned overlap, resulting in an image that can adequately protect privacy.

[0052] On the other hand, as shown in Figure 4, when the distance between the multi-pinhole mask 31 and the subject 4 is relatively close, the angle θ between the light incident on the first pinhole 33A and the light incident on the second pinhole 33B becomes larger, and the degree of overlap between the first subject image 4A and the second subject image 4B decreases. As a result, the image captured by the image sensor 32 becomes an image in which the subject 4 can be recognized, making it difficult to protect privacy.

[0053] When the distance between the multi-pinhole mask 31 and the subject 4 is relatively close, and the angle θ between the light incident on the first pinhole 33A and the light incident on the second pinhole 33B is large, the distance between the first subject image 4A and the second subject image 4B formed on the light-receiving surface of the image sensor 32 widens. As a result, the overlap between the first subject image 4A and the second subject image 4B decreases. Conversely, when the distance between the multi-pinhole mask 31 and the subject 4 is relatively far, and the angle θ between the light incident on the first pinhole 33A and the light incident on the second pinhole 33B is small, the distance between the first subject image 4A and the second subject image 4B formed on the light-receiving surface of the image sensor 32 narrows. As a result, the overlap between the first subject image 4A and the second subject image 4B increases.

[0054] In other words, even if the distance between the multi-pinhole mask 31 and the subject 4 is relatively close, by narrowing the distance between the first pinhole 33A and the second pinhole 33B, the angle θ between the light incident on the first pinhole 33A and the light incident on the second pinhole 33B becomes smaller, and the overlap between the first subject image 4A and the second subject image 4B can be increased.

[0055] Therefore, in this embodiment 1, the control device 2 controls at least one of the size and spacing of two of the multiple pinholes according to the distance between the subject in the image and the imaging device 3.

[0056] The control device 2 is specifically composed of a microprocessor, RAM (Random Access Memory), ROM (Read Only Memory), and a hard disk, which are not shown in the diagram. The RAM, ROM, or hard disk stores computer programs, and the functions of the control device 2 are realized when the microprocessor operates according to the computer programs.

[0057] The control device 2 includes an image acquisition unit 21, a recognition model storage unit 22, a control unit 23, a reference value storage unit 24, an output unit 25, and an image capture storage unit 26.

[0058] The image acquisition unit 21 acquires an image obtained by imaging with the imaging device 3, which is arranged such that a multi-pinhole mask 31 having multiple pinholes 33 covers the light-receiving surface of the image sensor 32. The image acquisition unit 21 outputs the acquired image to the image recognition unit 231 and output unit 25 of the control unit 23.

[0059] The recognition model storage unit 22 takes an image of the subject 4 obtained by the imaging device 3 as input and pre-stores a recognition model (estimation model) that has been trained using machine learning to output the distance between the subject 4 and the imaging device 3. The distance between the subject 4 and the imaging device 3 may be the distance between the subject 4 and the multi-pinhole mask 31, or the distance between the subject 4 and the light-receiving surface of the image sensor 32. The distance between the subject 4 and the multi-pinhole mask 31 may also be the length of the perpendicular line drawn from the subject 4 to the plane including the surface of the multi-pinhole mask 31. The distance between the subject 4 and the light-receiving surface of the image sensor 32 may also be the length of the perpendicular line drawn from the subject 4 to the plane including the light-receiving surface of the image sensor 32.

[0060] Furthermore, as a machine learning method, supervised learning is used, for example, which learns the relationship between input and output using training data in which labels (output information) are assigned to the input information. Other machine learning methods may include unsupervised learning, which constructs data structures from unlabeled inputs only; semi-supervised learning, which handles both labeled and unlabeled data; and reinforcement learning, which learns actions that maximize rewards through trial and error. Specific machine learning techniques include neural networks (including deep learning using multi-layered neural networks), genetic programming, decision trees, Bayesian networks, or support vector machines (SVMs). In the machine learning methods described in this disclosure, for example, deep neural networks (DNNs) or convolutional neural networks (CNNs) are used.

[0061] The control unit 23 controls the distance between two of the multiple pinholes (hereinafter also referred to as the pinhole width) based on the image acquired by the image acquisition unit 21. The control unit 23 calculates the distance between the subject 4 in the image and the imaging device 3, and controls the distance between the two pinholes based on the distance. The control unit 23 narrows the distance between the two pinholes as the distance between the subject 4 and the imaging device 3 decreases.

[0062] The control unit 23 includes an image recognition unit 231, a pinhole width calculation unit 232, and a pinhole control unit 233.

[0063] The image recognition unit 231 calculates the distance between the subject 4 in the image and the imaging device 3. The image recognition unit 231 calculates (estimates) the distance between the subject 4 and the imaging device 3 by inputting the image acquired by the image acquisition unit 21 into the recognition model read from the recognition model storage unit 22.

[0064] The reference value storage unit 24 stores in advance the reference pinhole width and reference distance used when calculating the pinhole width according to the distance between the subject 4 and the imaging device 3. The reference pinhole width and reference distance will now be explained.

[0065] Figure 5 is a schematic diagram illustrating the standard pinhole width and standard distance.

[0066] The pinhole width represents the distance between two of the multiple pinholes. For example, the pinhole width represents the distance between the center of the first pinhole 33A and the center of the second pinhole 33B. The reference pinhole width P is preset. The reference size (reference diameter or reference aperture area) of two of the multiple pinholes is also preset. The degree of overlap that provides sufficient privacy protection depends on the position and size of the pinholes, but privacy is considered to be protected when the subject image on the light-receiving surface of the image sensor 32 is superimposed with a predetermined degree of overlap. The predetermined degree of overlap is, for example, 50%.

[0067] The distance between subject 4 and the multi-pinhole mask 31 when the subject images overlap by 50% is set as the reference distance D. If the distance between the first pinhole 33A and the second pinhole 33B is the reference pinhole width P, then if the distance between subject 4 and the multi-pinhole mask 31 becomes longer than the reference distance D, the degree of overlap of the subject images will be greater than 50%. Conversely, if the distance between the first pinhole 33A and the second pinhole 33B is the reference pinhole width P, then if the distance between subject 4 and the multi-pinhole mask 31 becomes shorter than the reference distance D, the degree of overlap of the subject images will be less than 50%.

[0068] The reference value storage unit 24 pre-stores the reference pinhole width P and the reference distance D. The initial spacing between the first pinhole 33A and the second pinhole 33B is set to the reference pinhole width P. Furthermore, the reference value storage unit 24 also pre-stores the reference size of the pinholes. The initial size between the first pinhole 33A and the second pinhole 33B is set to the reference size.

[0069] If the multi-pinhole mask 31 has three or more pinholes, the distance between the pinholes with the largest spacing is set as the reference pinhole width P.

[0070] The pinhole width calculation unit 232 calculates the pinhole width for superimposing the subject image based on the distance between the subject 4 and the imaging device 3 calculated by the image recognition unit 231, and the reference pinhole width P and reference distance D stored in the reference value storage unit 24.

[0071] Figure 6 is a schematic diagram illustrating the method for calculating the pinhole width in this embodiment 1.

[0072] The reference pinhole width P, the reference distance D, and the reference angle θ formed by the light incident on the first pinhole 33A and the light incident on the second pinhole 33B have the following relationship:

[0073] tan(θ / 2)=(P / 2) / D=P / 2D atan(P / 2D)=θ / 2

[0074] When the estimated distance D’ (D’ < D) between the subject 4 and the imaging device 3 is calculated by the image recognition unit 231, the pinhole width P’ that satisfies the reference angle θ is expressed by the following formula.

[0075] atan(P’ / 2D’)=θ / 2 atan(P / 2D)=atan(P’ / 2D’) P / 2D=P’ / 2D’ P’=PD’ / D

[0076] Since the reference pinhole width P and the reference distance D are known, if the estimated distance D’ between the subject 4 and the imaging device 3 is calculated, the pinhole width P’, which is the distance between the first pinhole 33A’ and the second pinhole 33B’ that can protect privacy, is calculated. Therefore, the pinhole width calculation unit 232 multiplies the reference pinhole width P by the distance D’ between the subject 4 and the imaging device 3 calculated by the image recognition unit 231, and divides the multiplied value by the reference distance D to calculate the pinhole width P’.

[0077] When the multi-pinhole mask 31 has three or more pinholes, the pinhole width calculation unit 232 calculates the distance between the two pinholes with the largest interval.

[0078] The pinhole control unit 233 controls the distance between two of the plurality of pinholes. The pinhole control unit 233 changes the distance between the two pinholes so that the distance becomes the pinhole width calculated by the pinhole width calculation unit 232. When the calculated pinhole width is the same as the current pinhole width, the pinhole control unit 233 maintains the current distance between the two pinholes without changing the distance. When the calculated pinhole width is different from the current pinhole width, the pinhole control unit 233 changes the current pinhole width to the calculated pinhole width.

[0079] When changing the pinhole width, the pinhole control unit 233 changes the transmittance of the liquid crystal mask, the multi-pinhole mask 31, so as to form two pinholes of a predetermined size (reference size) at positions corresponding to the calculated pinhole width. The pinhole control unit 233 outputs a mask control signal to the multi-pinhole mask 31 to change the current pinhole width to the calculated pinhole width.

[0080] Furthermore, the pinhole control unit 233 outputs a control confirmation signal to the output unit 25 indicating whether or not the distance between the two pinholes has been controlled.

[0081] The output unit 25 outputs the image obtained by the imaging device 3 after the distance between the two pinholes has been controlled to the image storage unit 26. In other words, the output unit 25 stores the image obtained by the imaging device 3 after the distance between the two pinholes has been controlled in the image storage unit 26.

[0082] If a control confirmation signal is input indicating that the distance between the two pinholes has been controlled, the output unit 25 will not output the image obtained by the imaging device 3 to the image storage unit 26. In other words, if the distance between the two pinholes is changed, the image used for image recognition may have less blur and may not be able to protect privacy. Therefore, if the distance between the two pinholes is changed, the output unit 25 will discard the image obtained by the imaging device 3 without outputting it to the image storage unit 26.

[0083] On the other hand, if a control confirmation signal is input indicating that the distance between the two pinholes is not being controlled, the output unit 25 outputs the image obtained by the imaging device 3 to the image storage unit 26. In other words, if the distance between the two pinholes is not changed, the distance between the two pinholes has already been changed, so the image used for image recognition has an appropriate degree of blurring and can protect privacy. Therefore, if the distance between the two pinholes is not changed, the output unit 25 outputs the image obtained by the imaging device 3 to the image storage unit 26.

[0084] The image storage unit 26 stores the images obtained by the imaging device 3. At this time, the images stored in the image storage unit 26 are blurred images that can protect privacy.

[0085] In this embodiment 1, the control device 2 is equipped with an image storage unit 26, but the disclosure is not limited thereto, and an external computer connected to the control device 2 via a network may be equipped with an image storage unit 26. In this case, the output unit 25 may transmit the image obtained by the imaging device 3 to the external computer.

[0086] Next, the control processing in the control device 2 according to Embodiment 1 of this disclosure will be described.

[0087] Figure 7 is a flowchart illustrating the control process in the control device 2 according to Embodiment 1 of this disclosure. The flowchart shown in Figure 7 is executed at a predetermined sampling period. The predetermined sampling period is, for example, the frame period of the imaging device 3.

[0088] First, in step S1, the image acquisition unit 21 acquires an image obtained by imaging with the imaging device 3.

[0089] Next, in step S2, the image recognition unit 231 estimates the distance between the subject 4 in the image and the imaging device 3 by inputting the image acquired by the image acquisition unit 21 into the recognition model read from the recognition model storage unit 22. In this embodiment 1, the image recognition unit 231 estimates the distance between the subject 4 and the multi-pinhole mask 31 of the imaging device 3.

[0090] Next, in step S3, the pinhole width calculation unit 232 calculates the pinhole width for superimposing the subject image based on the distance estimated by the image recognition unit 231 and the reference pinhole width and reference distance stored in the reference value storage unit 24.

[0091] Next, in step S4, the pinhole control unit 233 determines whether the pinhole width calculated by the pinhole width calculation unit 232 is the same as the current pinhole width.

[0092] If it is determined that the calculated pinhole width is not the same as the current pinhole width (NO in step S4), then in step S5, the pinhole control unit 233 changes the current pinhole width to the calculated pinhole width. The pinhole control unit 233 outputs a mask control signal to the multi-pinhole mask 31 to change the current pinhole width to the calculated pinhole width. The multi-pinhole mask 31 changes the current pinhole width to the calculated pinhole width by changing the transmittance according to the input mask control signal. At this time, the pinhole control unit 233 outputs a control confirmation signal to the output unit 25 indicating that the pinhole width has been changed.

[0093] Next, in step S6, the output unit 25 discards the image obtained by the imaging device 3. When the output unit 25 receives a control confirmation signal indicating that the pinhole width has been changed, it discards the image obtained by the imaging device 3. After the processing in step S6 is completed, the process returns to step S1.

[0094] On the other hand, if it is determined that the calculated pinhole width is the same as the current pinhole width (YES in step S4), in step S7, the output unit 25 outputs the image obtained by the imaging device 3 to the image storage unit 26. When the processing in step S7 is completed, the process returns to step S1. As a result, the image obtained by the imaging device 3 is stored in the image storage unit 26. If it is determined that the calculated pinhole width is the same as the current pinhole width, the pinhole control unit 233 outputs a control confirmation signal to the output unit 25 indicating that the pinhole width was not changed. When the output unit 25 receives the control confirmation signal indicating that the pinhole width was not changed, it outputs the image obtained by the imaging device 3 to the image storage unit 26.

[0095] As described above, in the flowchart of Figure 7, after the processing of steps S6 and S7 is completed, the process returns to step S1. In step S1, the image acquisition unit 21 acquires a new image obtained from a new image taken by the imaging device 3. Then, using the new image acquired by the image acquisition unit 21, the processing from step S2 onwards shown in Figure 7 is performed again.

[0096] In this way, based on the image obtained by imaging with an imaging device 3, which is arranged so that a multi-pinhole mask 31 having multiple pinholes 33 covers the light-receiving surface of an image sensor 32, at least one of the size and spacing of two of the multiple pinholes 33 is controlled. Therefore, by controlling at least one of the size and spacing of two of the multiple pinholes 33 based on the image, an image is obtained that is difficult for humans to visually recognize as the subject, thus protecting the subject's privacy even when the distance from the imaging device 3 to the subject changes.

[0097] Furthermore, it is preferable that the frame rate of the imaging device 3 is faster than the normal frame rate of 30fps. For example, it is preferable that the frame rate of the imaging device 3 be 60fps. This allows for the collection of images obtained at the normal frame rate, even if an image that is not blurred is discarded. In addition, the image storage unit 26 can store moving images in which the privacy of the subject is protected.

[0098] In addition, in the first embodiment, a pinhole width in which the subject image satisfies a predetermined degree of overlap is calculated every time an image is acquired, but the present disclosure is not particularly limited thereto. When the distance between the subject 4 and the imaging device 3 calculated by the image recognition unit 231 is equal to or greater than a reference distance D stored in advance, it may not be necessary to change the pinhole width. That is, when the distance between the subject 4 and the imaging device 3 calculated by the image recognition unit 231 is shorter than the reference distance D stored in advance, the pinhole width may be calculated and changed. When the distance calculated in step S2 is shorter than the reference distance D, the processes of steps S3 to S7 may be performed. Further, when the distance calculated in step S2 is equal to or greater than the reference distance D, the processes of steps S3 to S6 may not be performed, and the process of step S7 may be performed.

[0099] Also, even when the distance between the subject 4 and the imaging device 3 is equal to or greater than the reference distance D and equal to or greater than a reference distance D2 (D < D2), the pinhole width may be changed. Specifically, when the distance between the subject 4 and the imaging device 3 is equal to or greater than the reference distance D and less than D2, the pinhole width is not changed, and when the distance between the subject 4 and the imaging device 3 is equal to or greater than the reference distance D2, the pinhole width is widened. This is because when the distance between the subject 4 and the imaging device 3 is too long, the degree of overlap of the subject image becomes too high, and conversely, the subject becomes easier to visually recognize.

[0100] In addition, when the pinhole width calculated by the pinhole width calculation unit 232 is equal to or less than the reference pinhole width, the pinhole control unit 233 may change the current pinhole width to the calculated pinhole width. Further, when the pinhole width calculated by the pinhole width calculation unit 232 is longer than the reference pinhole width, the pinhole control unit 233 may not change the current pinhole width to the calculated pinhole width.

[0101] Furthermore, in this embodiment 1, the distance between the subject 4 and the imaging device 3 is the distance between the subject 4 and the multi-pinhole mask 31, but this disclosure is not particularly limited thereto, and may be the distance between the subject 4 and the light-receiving surface of the image sensor 32. That is, when the distance between the subject 4 and the multi-pinhole mask 31 is used as training data during machine learning training of the recognition model, the image recognition unit 231 calculates the distance between the subject 4 and the multi-pinhole mask 31 from the image. On the other hand, when the distance between the subject 4 and the light-receiving surface of the image sensor 32 is used as training data during machine learning training of the recognition model, the image recognition unit 231 calculates the distance between the subject 4 and the light-receiving surface of the image sensor 32 from the image.

[0102] Furthermore, in this embodiment 1, the control unit 23 controls the spacing between two of the multiple pinholes based on the image acquired by the image acquisition unit 21, but this disclosure is not particularly limited thereto. The control unit 23 may also control the size of two of the multiple pinholes based on the image acquired by the image acquisition unit 21. The control unit 23 may also calculate the distance between the subject 4 in the image and the imaging device 3 and control the size of the two pinholes based on the distance. When the control unit 23 controls the size of the two pinholes, it increases the size as the distance between the subject 4 and the imaging device 3 decreases.

[0103] The control unit 23 may include a pinhole size calculation unit. The pinhole size calculation unit may calculate the pinhole size necessary to blur the subject image based on the distance between the subject 4 and the imaging device 3 calculated by the image recognition unit 231, and the reference size and reference distance stored in the reference value storage unit 24. The reference size is preset. The reference value storage unit 24 stores the reference size and reference distance.

[0104] The pinhole size calculation unit may calculate the standard size as the pinhole size if the calculated distance is greater than or equal to the standard distance. On the other hand, if the calculated distance is shorter than the standard distance, the pinhole size calculation unit may enlarge the standard size by an enlargement ratio corresponding to the calculated distance. For example, the enlargement ratio may be set to 1.0 when the calculated distance is the standard distance, and to 2.0 when the calculated distance is 0, and the pinhole size calculation unit may vary the enlargement ratio between 2.0 and 1.0 depending on the calculated distance.

[0105] Furthermore, the pinhole control unit 233 may control the size of two of the multiple pinholes. The pinhole control unit 233 changes the size of the two pinholes to match the size of the pinhole calculated by the pinhole size calculation unit. If the calculated size of the pinhole is the same as the current size of the pinhole, the pinhole control unit 233 may maintain the current size of the two pinholes without changing their sizes. If the calculated size of the pinhole is different from the current size of the pinhole, the pinhole control unit 233 may change the size of the current pinhole to match the calculated size of the pinhole.

[0106] Furthermore, when changing the size of the pinholes, the pinhole control unit 233 may change the transmittance of the liquid crystal mask, which is the multi-pinhole mask 31, to change the size of the two current pinholes to the calculated size. The pinhole control unit 233 may also output a mask control signal to the multi-pinhole mask 31 to change the size of the current pinholes to the calculated size.

[0107] Furthermore, the pinhole control unit 233 may output a control confirmation signal to the output unit 25 indicating whether or not the sizes of the two pinholes have been controlled.

[0108] The output unit 25 outputs the image obtained by the imaging device 3 after the sizes of the two pinholes have been controlled to the image storage unit 26. In other words, the output unit 25 may store the image obtained by the imaging device 3 after the sizes of the two pinholes have been controlled in the image storage unit 26.

[0109] If a control confirmation signal is input indicating that the sizes of the two pinholes have been controlled, the output unit 25 does not need to output the image obtained by the imaging device 3 to the image storage unit 26. If the sizes of the two pinholes are changed, the output unit 25 may discard the image obtained by the imaging device 3 without outputting it to the image storage unit 26.

[0110] On the other hand, if a control confirmation signal is input indicating that the sizes of the two pinholes are not being controlled, the output unit 25 may output the image obtained by the imaging device 3 to the image storage unit 26. If the sizes of the two pinholes are not changed, the output unit 25 may output the image obtained by the imaging device 3 to the image storage unit 26.

[0111] Furthermore, the control unit 23 may control both the size and spacing of two of the multiple pinholes based on the image acquired by the image acquisition unit 21. The control unit 23 may also calculate the distance between the subject 4 in the image and the imaging device 3, and control both the size and spacing of the two pinholes based on this distance.

[0112] (Embodiment 2) In Embodiment 1, the control device 2 superimposes the subject image by narrowing the distance between the two pinholes as the distance between the subject 4 in the image and the imaging device 3 decreases. However, as the distance between the two pinholes narrows, the degree of overlap between the two subject images increases, to the point where the two subject images essentially overlap into one, potentially making it possible for humans to visually recognize the subject.

[0113] Therefore, in the second embodiment, if the calculated distance between the two pinholes is shorter than a threshold, the control device controls the size of the two pinholes rather than controlling the distance between them.

[0114] Figure 8 is a block diagram showing an example of the overall configuration of the control system 1A in Embodiment 2 of this disclosure. In this Embodiment 2, the same reference numerals are used for components that are the same as in Embodiment 1, and their descriptions are omitted.

[0115] The control system 1A comprises a control device 2A and an imaging device 3. The control device 2A comprises an image acquisition unit 21, a recognition model storage unit 22A, a control unit 23A, a reference value storage unit 24, an output unit 25, and an image storage unit 26.

[0116] If multiple subjects 4 are visible in the image, the control unit 23A calculates the distance between each of the multiple subjects 4 and the imaging device 3, and controls at least one of the sizes and spacing of the two pinholes based on the distance between the subject 4 with the shortest calculated distance and the imaging device 3.

[0117] Furthermore, the control unit 23A calculates the distance between the two pinholes to be controlled based on the image. If the calculated distance is shorter than a threshold, it increases the size of the two pinholes beyond their current size. If the calculated distance is greater than or equal to the threshold, it narrows the distance between the two pinholes beyond their current size. In a pinhole camera, the larger the diameter of the pinhole, the more blurred the image becomes. Therefore, the control unit 23A obtains a blurred image that can protect privacy by increasing the size of the two pinholes beyond their current size.

[0118] The control unit 23A includes an image recognition unit 231A, a pinhole width calculation unit 232, a pinhole control unit 233A, and a subject selection unit 234.

[0119] The recognition model storage unit 22A takes an image of the subject 4 obtained by the imaging device 3 as input and pre-stores a recognition model (estimation model) that has been trained using machine learning to output the distance between the subject 4 and the imaging device 3 and the position of the subject 4 in the image.

[0120] The image recognition unit 231A calculates the distance between the subject 4 and the imaging device 3 by inputting the image acquired by the image acquisition unit 21 into the recognition model read from the recognition model storage unit 22A, and also detects the position of the subject 4 on the image. A This function estimates the distance between the subject 4 and the imaging device 3, as well as the position of the subject 4 in the image, from the image. The position of the subject 4 in the image is represented by a rectangular line surrounding the subject 4 in the image or by a line surrounding the outer edge of the subject 4 in the image.

[0121] Furthermore, if the image contains multiple subjects 4, the image recognition unit 231A estimates the position of each of the subjects 4, as well as the distance between each of the subjects 4 and the imaging device 3.

[0122] In this second embodiment, the recognition model storage unit 22A stores one recognition model for estimating distance and position, but the disclosure is not limited thereto. The recognition model storage unit 22A may store a first recognition model for estimating distance and a second recognition model for estimating position.

[0123] When the image recognition unit 231A estimates the positions of multiple subjects 4 on an image, the subject selection unit 234 selects the subject 4 with the shortest calculated distance from among the multiple subjects 4 on the image. In other words, if multiple subjects 4 are present in the image, the closest subject 4 is selected from among the multiple subjects 4.

[0124] The pinhole width calculation unit 232 calculates the distance between the two pinholes (pinhole width) based on the shortest calculated distance between the subject 4 and the imaging device 3.

[0125] The pinhole control unit 233A determines whether the pinhole width calculated by the pinhole width calculation unit 232 is shorter than a threshold. The threshold is an interval at which it becomes difficult for a human to visually recognize an object, for example, an interval at which the degree of overlap between two object images on the light-receiving surface of the image sensor 32 is 90%.

[0126] If the calculated pinhole width is shorter than the threshold, the pinhole control unit 233A changes the size of the two pinholes to be larger than their current size. In this case, the pinhole control unit 233A enlarges the size of the two pinholes by a predetermined magnification factor. If the calculated pinhole width is greater than or equal to the threshold, the pinhole control unit 233A changes the current pinhole width to the calculated pinhole width.

[0127] The output unit 25 outputs the image obtained by the imaging device 3 after the spacing or size of the two pinholes has been controlled to the image storage unit 26. In other words, the output unit 25 stores the image obtained by the imaging device 3 after the spacing or size of the two pinholes has been controlled in the image storage unit 26.

[0128] Next, the control processing in the control device 2A according to Embodiment 2 of this disclosure will be described.

[0129] Figure 9 is a flowchart illustrating the control process in the control device 2A according to Embodiment 2 of this disclosure. The flowchart shown in Figure 9 is executed at a predetermined sampling period. The predetermined sampling period is, for example, the frame period of the imaging device 3.

[0130] First, in step S11, the image acquisition unit 21 acquires an image obtained by imaging with the imaging device 3.

[0131] Next, in step S12, the image recognition unit 231A controls the recognition model storage unit 22 ABy inputting the image acquired by the image acquisition unit 21 into the recognition model read from the image, the distance between the subject 4 in the image and the imaging device 3, and the position of the subject 4 in the image are estimated.

[0132] Next, in step S13, the subject selection unit 234 determines whether or not there are multiple subjects in the image based on the position of subject 4 in the image estimated by the image recognition unit 231A. If it is determined that there are no multiple subjects in the image (NO in step S13), the process proceeds to step S15.

[0133] On the other hand, if it is determined that there are multiple subjects in the image (YES in step S13), in step S14, the subject selection unit 234 selects the subject 4 from among the multiple subjects 4 in the image that has the shortest distance estimated by the image recognition unit 231A.

[0134] Next, in step S15, the pinhole width calculation unit 232 calculates the pinhole width for superimposing the subject image based on the distance estimated by the image recognition unit 231A and the reference pinhole width and reference distance stored in the reference value storage unit 24. At this time, if there is only one subject on the image, the pinhole width calculation unit 232 calculates the pinhole width for superimposing the subject image based on the distance between the one subject and the imaging device 3 and the reference pinhole width and reference distance stored in the reference value storage unit 24. On the other hand, if there are multiple subjects on the image, the pinhole width calculation unit 232 calculates the pinhole width for superimposing the subject images based on the distance between the subject with the shortest estimated distance and the imaging device 3 and the reference pinhole width and reference distance stored in the reference value storage unit 24.

[0135] Next, in step S16, the pinhole control unit 233A determines whether the pinhole width calculated by the pinhole width calculation unit 232 is the same as the current pinhole width.

[0136] If it is determined that the calculated pinhole width is the same as the current pinhole width (YES in step S16), then in step S22, the output unit 25 outputs the image obtained by the imaging device 3 to the image storage unit 26. After the processing in step S22 is completed, the process returns to step S11. As a result, the image obtained by the imaging device 3 is stored in the image storage unit 26.

[0137] On the other hand, if it is determined that the calculated pinhole width is not the same as the current pinhole width (NO in step S16), in step S17, the pinhole control unit 233A determines whether the pinhole width calculated by the pinhole width calculation unit 232 is shorter than a threshold.

[0138] If the calculated pinhole width is determined to be shorter than the threshold (YES in step S17), then in step S18, the pinhole control unit 233A determines whether the sizes of the two pinholes have been changed from the reference size.

[0139] If it is determined that the sizes of the two pinholes have been changed from the standard size (YES in step S18), then in step S22, the output unit 25 outputs the image obtained by the imaging device 3 to the image storage unit 26.

[0140] On the other hand, if it is determined that the size of the two pinholes has not changed from the reference size (NO in step S18), in step S19, the pinhole control unit 233A changes the size of the two pinholes to be larger than their current size. After the processing in step S19 is completed, the process returns to step S11. The pinhole control unit 233A outputs a mask control signal to the multi-pinhole mask 31 to enlarge the size of the two pinholes by a predetermined magnification while maintaining the current pinhole width. The multi-pinhole mask 31 changes the size of the two pinholes by changing the transmittance according to the input mask control signal. At this time, the pinhole control unit 233A outputs a control confirmation signal to the output unit 25 indicating that the size of the two pinholes has been changed.

[0141] Furthermore, if it is determined that the calculated pinhole width is greater than or equal to a threshold (NO in step S17), in step S20, the pinhole control unit 233A changes the current pinhole width to the calculated pinhole width. At this time, if the sizes of the two pinholes have been changed from the standard size, the pinhole control unit 233A changes the sizes of the two pinholes to the standard size and changes the current pinhole width to the calculated pinhole width.

[0142] Note that the processes in steps S20 and S21 are the same as those in steps S5 and S6 shown in Figure 7, so their explanation is omitted. Once the process in step S21 is completed, the process returns to step S11.

[0143] As described above, in the flowchart of Figure 9, after the processing of steps S19, S21, and S22 is completed, the process returns to step S11. In step S11, the image acquisition unit 21 acquires a new image obtained from a new image taken by the imaging device 3. Then, using the new image acquired by the image acquisition unit 21, the processing from step S12 onwards shown in Figure 9 is performed again.

[0144] As described above, as the distance between the two pinholes 33 narrows, the degree of overlap between the two subject images on the light-receiving surface of the image sensor 32 increases, and the two subject images may effectively overlap into one, potentially making it possible for humans to visually recognize the subject. However, according to this second embodiment, if the calculated pinhole width is shorter than the threshold, the pinhole width is not controlled; instead, the size of the two pinholes is controlled to be larger than their current size. Therefore, even if the degree of overlap between the two subject images on the light-receiving surface of the image sensor 32 increases, an image can be obtained that makes it difficult for humans to visually recognize the subject.

[0145] In this embodiment 2, the pinhole control unit 233A enlarges the size of the two pinholes at a predetermined magnification ratio, but the disclosure is not limited thereto, and the current size of the two pinholes may be changed to a predetermined size that is larger than the reference size.

[0146] Furthermore, the pinhole control unit 233A may determine a magnification factor according to the difference between the calculated pinhole width and the current pinhole width, and enlarge the size of the two pinholes by the determined magnification factor. In this case, the pinhole control unit 233A may increase the magnification factor as the difference value increases.

[0147] Furthermore, in Embodiment 1, the control unit 23 may, similar to Embodiment 2, calculate the distance between each of the multiple subjects 4 and the imaging device 3 when multiple subjects 4 are visible in the image, and control at least one of the sizes and spacing of the two pinholes based on the distance between the subject 4 with the shortest calculated distance and the imaging device 3.

[0148] (Embodiment 3) In Embodiment 1, the control device 2 superimposes the subject image by narrowing the pinhole width as the distance between the subject 4 in the image and the imaging device 3 decreases. However, even when the pinhole width is narrowed, it is sometimes possible to obtain an image with a low degree of blurring that allows humans to visually recognize the subject.

[0149] Therefore, in Embodiment 3, the control device controls one of the two pinhole sizes and spacings, and if the degree of blurring in the image acquired by the image acquisition unit 21 is below a threshold, it controls the uncontrolled pinhole size and spacing.

[0150] Figure 10 is a block diagram showing an example of the overall configuration of the control system 1B in Embodiment 3 of this disclosure. In this Embodiment 3, the same reference numerals are used for components that are the same as those in Embodiments 1 and 2, and their descriptions are omitted.

[0151] The control system 1B comprises a control device 2B and an imaging device 3. The control device 2B comprises an image acquisition unit 21, a recognition model storage unit 22B, a control unit 23B, a reference value storage unit 24, an output unit 25, and an image storage unit 26.

[0152] If multiple subjects 4 are visible in the image, the control unit 23B calculates the distance between each of the multiple subjects 4 and the imaging device 3, and controls at least one of the sizes and spacing of the two pinholes based on the distance between the subject 4 with the shortest calculated distance and the imaging device 3.

[0153] Furthermore, the control unit 23B calculates the degree of blur of the subject 4 in the image and controls at least one of the size and spacing of the two pinholes based on the degree of blur. After controlling one of the size and spacing of the two pinholes, the control unit 23B calculates the degree of blur of the subject 4 in the image, and if the degree of blur is below a threshold, controls the uncontrolled size and spacing of the two pinholes. That is, after controlling the spacing of the two pinholes, the control unit 23B calculates the degree of blur of the subject 4 in the image, and if the degree of blur is below a threshold, controls the size of the two pinholes. Alternatively, the control unit 23B may calculate the degree of blur of the subject 4 in the image after controlling the size of the two pinholes, and if the degree of blur is below a threshold, control the spacing of the two pinholes. The control unit 23B calculates the degree of blur by inputting the acquired image into a recognition model trained using machine learning to take an image containing the subject 4 as input and output the degree of blur of the subject in the image.

[0154] As described above, with a pinhole camera, the larger the diameter of the pinhole, the more blurred the image becomes. Therefore, if the control unit 23B cannot obtain a sufficiently blurred image by changing the distance between the two pinholes, it increases the size of the pinholes beyond the current size to obtain a blurred image that protects privacy.

[0155] The control unit 23B includes an image recognition unit 231B, a pinhole width calculation unit 232, a pinhole control unit 233B, a subject selection unit 234B, and a blur degree determination unit 235.

[0156] The recognition model storage unit 22B takes an image of the subject 4 obtained by the imaging device 3 as input and pre-stores a recognition model (estimation model) that has been trained using machine learning to output the distance between the subject 4 and the imaging device 3, the position of the subject 4 in the image, and the degree of blur.

[0157] The image recognition unit 231B calculates the distance between the subject 4 and the imaging device 3 by inputting the image acquired by the image acquisition unit 21 into the recognition model read from the recognition model storage unit 22B, detects the position of the subject 4 in the image, and calculates the degree of blur of the subject 4 in the image. B This system estimates the distance between the subject 4 and the imaging device 3, the position of the subject 4 in the image, and the degree of blurring of the subject 4 in the image from the image. The position of the subject 4 in the image is represented by a rectangular line surrounding the subject 4 in the image or by a line surrounding the outer edge of the subject 4 in the image. The degree of blurring is expressed, for example, as 0 to 100%, and is an indicator of how blurred the subject 4 is in the image. The higher the degree of blurring, the more difficult it is for a human to visually recognize the subject.

[0158] Furthermore, if the image contains multiple subjects 4, the image recognition unit 231B estimates the position of each of the subjects 4, the distance between each of the subjects 4 and the imaging device 3, and the degree of blurring for each of the subjects 4.

[0159] In this third embodiment, the recognition model storage unit 22B stores one recognition model for estimating distance, position, and degree of blur, but the disclosure is not limited thereto. The recognition model storage unit 22B may store a first recognition model for estimating distance, a second recognition model for estimating position, and a third recognition model for estimating degree of blur.

[0160] The blur degree determination unit 235 determines whether the pinhole width was changed in the previous instance. If the blur degree determination unit 235 determines that the pinhole width was changed in the previous instance, it determines whether the blur degree calculated by the image recognition unit 231B is below a threshold. The threshold is the blur degree used to determine whether or not a human can visually recognize the subject. If the blur degree calculated by the image recognition unit 231B is below the threshold, a human can visually recognize the subject. If the blur degree calculated by the image recognition unit 231B is higher than the threshold, it is difficult for a human to visually recognize the subject.

[0161] The subject selection unit 234B is determined by the degree of blur determination unit 235 to have not changed the pinhole width in the previous instance, and the image recognition unit 231 B If the positions of multiple subjects 4 on the image are estimated by this, the subject 4 with the shortest calculated distance is selected from among the multiple subjects 4 on the image. B This occurs when the blur degree determination unit 235 determines that the blur degree is higher than the threshold, and the image recognition unit 231 B If the positions of multiple subjects 4 are estimated on the image, the subject 4 with the shortest calculated distance is selected from among the multiple subjects 4 on the image. In other words, if multiple subjects 4 are present in the image, the closest subject 4 is selected from among the multiple subjects 4.

[0162] If the blur level determination unit 235 determines that the blur level is below a threshold, the pinhole control unit 233B changes the size of the two pinholes to be larger than their current size. In this case, the pinhole control unit 233B enlarges the size of the two pinholes at a predetermined magnification ratio. Furthermore, if the blur level determination unit 235 determines that the blur level is higher than a threshold, the pinhole control unit 233B changes the current pinhole width to the calculated pinhole width.

[0163] The pinhole control unit 233B turns on the pinhole width control flag if it controls the distance between the two pinholes, and turns off the pinhole width control flag if it does not control the distance between the two pinholes. The blur degree determination unit 235 determines whether the pinhole width was changed last time by referring to the pinhole width control flag.

[0164] The output unit 25 outputs the image obtained by the imaging device 3 after the spacing or size of the two pinholes has been controlled to the image storage unit 26. In other words, the output unit 25 stores the image obtained by the imaging device 3 after the spacing or size of the two pinholes has been controlled in the image storage unit 26.

[0165] Next, the control processing in the control device 2B according to Embodiment 3 of this disclosure will be described.

[0166] Figure 11 is a flowchart illustrating the control process in the control device 2B according to Embodiment 3 of this disclosure. The flowchart shown in Figure 11 is executed at a predetermined sampling period. The predetermined sampling period is, for example, the frame period of the imaging device 3.

[0167] First, in step S31, the image acquisition unit 21 acquires an image obtained by imaging with the imaging device 3.

[0168] Next, in step S32, the image recognition unit 231B inputs the image acquired by the image acquisition unit 21 into the recognition model read from the recognition model storage unit 22, thereby estimating the distance between the subject 4 in the image and the imaging device 3, the position of the subject 4 in the image, and the degree of blurring of the subject 4 in the image.

[0169] Next, in step S33, the blur degree determination unit 235 determines whether the pinhole width was changed in the previous step. If it is determined that the pinhole width was changed in the previous step (YES in step S33), then in step S34, the blur degree determination unit 235 determines whether the blur degree estimated by the image recognition unit 231B is below a threshold.

[0170] If the degree of blurring is determined to be below a threshold (YES in step S34), then in step S35, the pinhole control unit 233B changes the size of the two pinholes to be larger than their current size. The pinhole control unit 233B outputs a mask control signal to the multi-pinhole mask 31 to enlarge the two pinholes at a predetermined magnification while maintaining the current pinhole width. The multi-pinhole mask 31 changes the size of the two pinholes by changing the transmittance according to the input mask control signal. At this time, the pinhole control unit 233 BThe system outputs a control confirmation signal to the output unit 25 indicating that the sizes of the two pinholes have been changed. After the processing in step S35 is completed, the system returns to the processing in step S31.

[0171] The blur degree determination unit 235, when there are multiple subjects in the image, determines whether the blur degree of each subject is below a threshold. If the blur degree of at least one subject is below the threshold, the process proceeds to step S35; if the blur degree of all subjects is above the threshold, the process proceeds to step S36.

[0172] On the other hand, if it is determined that the pinhole width has not been changed in the previous step (NO in step S33), or if it is determined that the degree of blur is higher than the threshold (NO in step S34), then in step S36, the subject selection unit 234 B Based on the position of subject 4 in the image estimated by the image recognition unit 231B, it determines whether or not there are multiple subjects in the image.

[0173] Note that the processes in steps S36 to S39 are the same as those in steps S13 to S16 shown in Figure 9, so their explanation will be omitted.

[0174] If it is determined that the calculated pinhole width is not the same as the current pinhole width (NO in step S39), in step S40, the pinhole control unit 233B changes the current pinhole width to the calculated pinhole width. At this time, if the sizes of the two pinholes have been changed from the standard size, the pinhole control unit 233B changes the sizes of the two pinholes to the standard size and changes the current pinhole width to the calculated pinhole width.

[0175] Note that the processes in steps S40 to S42 are the same as those in steps S5 to S7 shown in Figure 7, so their explanation is omitted. After the process in step S41 is completed, the process returns to step S31, and after the process in step S42 is completed, the process returns to step S31.

[0176] As described above, in the flowchart of Figure 11, after the processing of steps S35, S41, and S42 is completed, the process returns to step S31. In step S31, the image acquisition unit 21 acquires a new image obtained by the imaging device 3's new imaging. Then, using the new image acquired by the image acquisition unit 21, the processing from step S32 onwards shown in Figure 11 is performed again.

[0177] According to this third embodiment, even if the degree of blurring of the subject in the image is below a threshold after one of the sizes and spacing of the two pinholes 33 is controlled, the uncontrolled size and spacing of the two pinholes 33 is controlled, thereby increasing the degree of blurring of the subject in the image and making it difficult for humans to visually recognize the subject.

[0178] In this embodiment 3, the pinhole control unit 233B enlarges the size of the two pinholes at a predetermined magnification ratio, but the disclosure is not limited thereto, and the current size of the two pinholes may be changed to a predetermined size that is larger than the reference size. [Industrial applicability]

[0179] The technology described herein can protect the privacy of a subject even when the distance from the imaging device to the subject changes, and is therefore useful for technology that controls an imaging device in which a mask having multiple pinholes is arranged to cover the light-receiving surface of an image sensor.

Claims

1. An acquisition unit that acquires an image obtained by imaging of an imaging device in which a mask having multiple pinholes is arranged to cover the light-receiving surface of an image sensor, A control unit that controls at least one of the size and spacing of the two pinholes with the largest spacing among the plurality of pinholes based on the aforementioned image, An output unit that outputs an image obtained by the imaging device after at least one of the size and spacing of the two pinholes has been controlled, A control device equipped with the following features.

2. The control unit calculates the distance between the subject in the acquired image and the imaging device based on the acquired image, and controls at least one of the sizes and spacing of the two pinholes based on the distance. The control device according to claim 1.

3. The control unit takes the acquired image showing the subject as input and calculates the distance by inputting the acquired image into an estimation model trained using machine learning to output the distance between the subject and the imaging device. The control device according to claim 2.

4. When the control unit controls the size, it increases the size as the distance decreases, and when the control unit controls the interval, it narrows the interval as the distance decreases. The control device according to claim 2 or 3.

5. The control unit calculates the distance to be controlled between the two pinholes based on the acquired image, and if the calculated distance is shorter than a threshold, it increases the size, and if the calculated distance is greater than or equal to the threshold, it narrows the distance. The control device according to claim 1.

6. The control unit calculates the degree of blurring of the subject in the acquired image and controls at least one of the size and spacing of the two pinholes based on the degree of blurring. The control device according to claim 1.

7. The control unit controls one of the two pinhole sizes and spacings, then calculates the degree of blurring of the subject in the acquired image, and if the degree of blurring is below a threshold, controls the uncontrolled of the two pinhole sizes and spacings. The control device according to claim 1.

8. The control unit takes the acquired image in which the subject is shown as input and calculates the degree of blur by inputting the acquired image into an estimation model trained using machine learning to output the degree of blur of the subject shown in the acquired image. The control device according to claim 6 or 7.

9. If the acquired image contains multiple subjects, the control unit calculates the distance between each of the multiple subjects and the imaging device based on the acquired image, and controls at least one of the sizes and spacings of the two pinholes based on the distance between the subject with the shortest calculated distance and the imaging device. The control device according to any one of claims 1 to 8.

10. The image output by the output unit is a new image obtained by a new image captured by the imaging device after the control unit has controlled at least one of the size and spacing of the two pinholes. The control device according to claim 1.

11. Computers An image obtained by imaging with an imaging device in which a mask having multiple pinholes is arranged to cover the light-receiving surface of the image sensor is acquired. Based on the aforementioned image, control at least one of the size and spacing of the two pinholes with the largest spacing among the plurality of pinholes. The image obtained by the imaging device after at least one of the size and spacing of the two pinholes is controlled is output. Control method.

Images