imaging device

Abstract

The imaging device is provided with an imaging element that acquires a captured image, and a protection member that is a member that covers at least an imaging surface of the imaging element and protects the imaging element, and is configured to guide light from the outside to the imaging surface through a hole that penetrates in a thickness direction.

Description

Technical Field

[0001] This technology relates to imaging devices, and more specifically to the field of imaging devices having protective functions for protecting the interior from external influences. Background Technology

[0002] For example, with the miniaturization of imaging devices such as digital still cameras and digital video cameras, the demand for quality maintenance in previously less-considered usage environments and applications has increased as devices become smaller and their applications expand.

[0003] For example, countermeasures need to be taken for the following situations: (a) Distortion of the optical system caused by thermal expansion and contraction, vibration, etc., in harsh natural environments (high mountains, polar regions, deep sea, volcanoes, etc.). (b) Surface corrosion caused by volcanic gases and liquids and gases inside the plant. (c) “Electronic component failure” such as that of sensors caused by strong electromagnetic fields around high-voltage transmission lines. (d) Damage to electronic components such as sensors caused by radiation around nuclear power plants and heavy particle beams from outer space. (e) “Accumulation of dirt” due to the routine use of outdoor (surveillance cameras, cameras mounted on mobile devices, etc.).

[0004] Note that Non-Patent Document 1 described below may be cited as relevant prior art. Non-Patent Document 1 discloses a technique for an imaging device with a so-called lensless camera, which receives light incident through a micro-aperture as a pinhole to obtain a captured image. This technique uses an artificial intelligence (AI) model to correct for image quality degradation caused by the absence of a lens.

[0005] Reference List

[0006] Non-patent literature

[0007] Non-patent document 1: "Image reconstruction with transformer for mask-basedlensless imaging": Xiuxi Pan, Xiao Chen, Saori Takeyama, Masahiro Yamaguchi (Optics Letters Vol. 47, Issue 7, pp. 1843-1846 (2022)) Summary of the Invention

[0008] The problem to be solved by the present invention

[0009] Regarding the issues in (a) to (e) above, the following approach may be considered: (a) Countermeasures using heat-resistant materials and vibration-absorbing materials (non-resonant materials), (b) Countermeasures using corrosion-inhibiting materials corresponding to the target chemical substance, (c) Countermeasures using metal shielding or the like for electromagnetic shielding, (d) Countermeasures using heavy metals (lead, etc.) to provide radiation shielding, and (e) Countermeasures using hydrophobic materials or coatings with hydrophobic layers.

[0010] However, ensuring protective functions presents a challenge for the lens portion of the imaging device. Lenses made of materials such as glass, fluorite, or plastic are embedded in this portion, and furthermore, the lenses need to be transparent and have a special shape for refraction and image formation.

[0011] Protective components formed on the outer casing cannot necessarily be implemented using transparent materials such as lenses. Furthermore, even if they could be implemented using transparent materials, it is not necessarily possible to form the transparent material into a lens shape, or to cover or coat a lens.

[0012] Therefore, the combined use of the lens and protective components is difficult in most cases.

[0013] In view of the above, this technology is proposed, and its purpose is to appropriately balance imaging function and protection function for internal components of the device.

[0014] Solution to the problem

[0015] An imaging apparatus according to the present technology includes an imaging element for acquiring a captured image and a protective member covering at least an imaging surface of the imaging element to protect the imaging element, the protective member being configured to guide light from the outside to the imaging surface through a hole extending in the thickness direction.

[0016] According to the above structure, it is possible to acquire captured images through the same mechanism as a pinhole camera, while at least protecting the imaging element inside the device. Attached Figure Description

[0017] [ Figure 1 [This is an external perspective view of an imaging device according to a first embodiment of the present technology.]

[0018] [ Figure 2 [ ] is a cross-sectional view of the imaging device as a first embodiment.

[0019] [ Figure 3 [Illustration of an image obtained by an imaging device that functions as a pinhole camera]

[0020] [ Figure 4 [This is an external perspective view of an imaging device with multiple holes.]

[0021] [ Figure 5 [This is a cross-sectional view of an imaging device with multiple holes.]

[0022] [ Figure 6 [ ] is a diagram illustrating the cross-sectional structure of an imaging device when a protective component is used as a heat dissipation component.

[0023] [ Figure 7 [ ] is a diagram illustrating the cross-sectional structure of an imaging device when a protective component is used as a hydrophobic component.

[0024] [ Figure 8 [ ] is a block diagram showing an example of the internal configuration of an imaging device as a first embodiment.

[0025] [ Figure 9 [This is a diagram showing the superposition of position offsets when using a multi-pinhole method.]

[0026] [ Figure 10 [Illustration] is an example of a hole formation pattern when using a multi-pinhole method.

[0027] [ Figure 11 [Is adopting] Figure 10 An illustrative diagram of a blurred image obtained when a pattern is formed, as shown.

[0028] [ Figure 12 [This is an illustration of the learning method used when an AI model is used for fuzz correction.]

[0029] [ Figure 13 [Illustration] is a diagram illustrating an image transformation function model that uses GMLP as the encoder and FNN as the decoder.

[0030] [ Figure 14 [This is a schematic diagram illustrating the path of light from a specific point on the subject when the aperture has thickness.]

[0031] [ Figure 15 [ ] is an explanatory diagram of the reflection suppression component.

[0032] [ Figure 16 [This is an explanatory diagram of the subject's illumination range and the imageable area (imaging angle).]

[0033] [ Figure 17 [This is an illustration of the path of light rays emanating from the outermost peripheral aperture of the subject.]

[0034] [ Figure 18 [This is an illustration of the path of light rays that can pass through the outermost periphery of the subject through the outermost aperture.]

[0035] [ Figure 19 [ ] is a block diagram showing an example of the internal configuration of an imaging device as a second embodiment.

[0036] [ Figure 20 [This is a schematic diagram showing the area of ​​transmitted light.]

[0037] [ Figure 21 [This is a diagram used to illustrate the hole-changing mechanism of the rotary table method.]

[0038] [ Figure 22 [ ] is a block diagram showing an example of the internal configuration of an imaging device as a third embodiment.

[0039] [ Figure 23 [This is a diagram used to illustrate the sliding hole changing mechanism.] Detailed Implementation

[0040] In the following text, embodiments according to this technology will be described in the following order.

[0041] <1. First Implementation Method>

[0042] [1-1. Example of an imaging device structure]

[0043] [1-2. Regarding protection functions]

[0044] [1-3. About Image Correction]

[0045] [1-4. Solutions to problems caused by hole thickness]

[0046] <2. Second Implementation Method>

[0047] <3. Third Implementation Method>

[0048] <4. Applications of Imaging Devices>

[0049] <5. Variations>

[0050] <6. Summary of Implementation Methods>

[0051] <7. This technology>

[0052] <1. First Implementation Method>

[0053] [1-1. Example of an imaging device structure]

[0054] Figure 1 and Figure 2 This is a diagram illustrating an example of the structure of an imaging apparatus 1 according to a first embodiment of the present technology, wherein, Figure 1 This is an external perspective view of imaging device 1, and Figure 2 This is a cross-sectional view of imaging device 1.

[0055] The imaging device 1 has the following configuration: the imaging element 2 is arranged inside the outer housing, and the protective component 3 for protecting the internal components of the device including the imaging element 2 is used as the outer housing.

[0056] Furthermore, the specific protective functions of the protective component 3 will be described later.

[0057] Imaging device 1 is configured as a digital camera, such as a digital still camera or a digital video camera, and an image sensor that acquires captured images through digital data is used as imaging element 2.

[0058] like Figure 2 As shown, a circuit unit 4, in addition to the imaging element 2, is formed inside the imaging device 1. The circuit unit 4 includes electronic circuitry for processing the captured image obtained by the imaging element 2, and circuitry for supplying power from a power supply unit such as a battery (not shown).

[0059] Figure 1 and Figure 2 An example of an imaging device 1 configured as a so-called pinhole camera is shown, in which a single hole 3a extending through the thickness direction is formed in the protective member 3, and light from the outside is guided to the imaging surface through the hole 3a.

[0060] In this case, the size of aperture 3a is sufficiently smaller than the size and scale of the subject to be imaged, and aperture 3a is formed as a micro-aperture, for example, less than 100 micrometers square.

[0061] In the following text, the portion of the protective member 3 with the hole 3a will be referred to as the light-transmitting portion 31.

[0062] Figure 3 This is an explanatory diagram of an image obtained by an imaging device 1, which is a pinhole camera.

[0063] As shown in the figure, in the imaging device 1, which is a pinhole camera, light from each point of the subject passes through a single aperture 3a at a different incident angle and illuminates the imaging surface of the imaging element 2. In this case, as shown in the figure, an inverted real image is obtained as an image formed on the imaging surface, and a captured image is obtained as an image obtained by capturing the inverted real image.

[0064] In the imaging device 1 that captures images using the principle of a pinhole camera as described above, the area required for imaging is only the micro-hole 3a, which serves as the pinhole, and the area that cannot be covered by the protective component 3 is significantly narrower compared to the lens portion of a conventional camera.

[0065] While maintaining the required function of the lens portion through the hole 3a, various factors that could degrade quality can be avoided through the lower passage of the protective component 3, which will be described later.

[0066] In addition, in the protective component 3, the parts other than the hole 3a are required to be opaque. If the substrate is transparent, an opaque material can also be covered or coated.

[0067] Here, pinhole cameras tend to produce less light compared to cameras with lenses that utilize light-gathering effects. Therefore, they are not suitable for imaging in low light or for high-speed imaging.

[0068] Therefore, to increase the amount of light, an imaging device 1 can also be considered, wherein multiple holes 3a are formed in the protective component 3, such as... Figure 4 External perspective and Figure 5 The cross-sectional view is shown below.

[0069] Note that in order to increase the amount of light, the size of the hole 3a, which serves as the pinhole, can also be increased.

[0070] like Figure 1 Hole 3a in the image is a single pinhole camera, which is known to produce blur in the captured image.

[0071] Moreover, in adopting such Figure 4 In the case of multiple apertures 3a or an increased size of aperture 3a to increase light intensity, the blurriness of the captured image increases.

[0072] Note that countermeasures for such blurred images will be described later.

[0073] [1-2. Regarding protection functions]

[0074] The protective function of protective component 3 will be described.

[0075] As a protective component 3, a shielding component or a discharge and circulation component can be considered. The shielding component isolates the inside of the device from the outside, while the discharge and circulation component has the function of external discharge or internal circulation to improve the internal environment of the device.

[0076] Examples of shielding components include electromagnetic shielding components, radiation shielding components, shock absorbing components, vibration absorbing components, heat-resistant components, corrosion-inhibiting components, and bio-repellent components. Additionally, shielding components include hydrophobic components.

[0077] Examples of exhaust and circulation components include heat dissipation components, ventilation components, humidification components, drying components, etc.

[0078] By using these shielding components, as well as discharge and circulation components, as protective components, the internal components of the device can be appropriately protected according to the assumed use cases.

[0079] (Example of electromagnetic shielding components)

[0080] Examples of electromagnetic shielding components include electromagnetic wave shielding components and electromagnetic field shielding components.

[0081] Electromagnetic waves are classified according to their wavelength into gamma rays, X-rays, ultraviolet rays (UVA, UVB, and UVC), visible light, infrared rays (NIR: near-infrared and FIR: far-infrared), terahertz light, microwaves, millimeter waves, etc. Among these rays, some are known to cause different types of problems in electronic devices, living organisms, etc.

[0082] Electromagnetic wave shielding usually refers to microwave shielding. It is known that microwaves can have adverse effects on living organisms due to the amplification of water molecule vibrations, and can also cause circuit failures by inducing electronic noise, as well as damage to circuit boards due to the photoelectric effect.

[0083] Electromagnetic shielding utilizes the following of free electrons in a metal to the electromagnetic field to cancel electromagnetic waves, and simply involves covering the area around the object to be shielded with metal. Although some gaps exist, metal shielding can function even with gaps, but since the light transmission part 31 is directly connected to sensor circuits, it is desirable to shield the light transmission part as much as possible.

[0084] For example, when the area of ​​the aperture 3a in the light-passing section 31 is small, a bulk metal can be used. If weight reduction is required, it can be formed from a wire mesh. When using a wire mesh, it is possible to embed the wire mesh into an opaque material, making the area outside the aperture 3a opaque.

[0085] Metal shielding is also effective for shielding alternating electric and magnetic fields.

[0086] On the other hand, in cases where there is a risk of leakage or electric shock due to power supply, measures such as covering the metal shield with an insulating coating should be considered.

[0087] (Example of a radiation shield)

[0088] X-rays and gamma rays have high light energy, and these rays can damage electronic components with microstructures due to photoelectric effects, etc. Thick metal plates of lead or tungsten are typically used to shield X-rays, gamma rays, and other radiation. Similar to microwaves, when the light passage 31 is not shielded, there is a possibility that radiation can easily reach the internal circuitry and damage electronic components; therefore, it is desirable to also use lead or tungsten to form the light passage 31.

[0089] Here, environmentally resistant materials, such as electromagnetic shielding, typically have the effect of attenuating the intensity of the physical phenomenon to be protected through an exponential function of thickness. When the intensity of the physical phenomenon on the outer surface of the shielding component is I0, the intensity I(t) across the material thickness t can be expressed as follows.

[0090] I(t) = I0exp(-γt)

[0091] However, note that γ is the attenuation coefficient, which is a constant determined by the combination of shielding material and physical phenomena, and its reciprocal is a quantity with distance dimensions, such as attenuation distance or attenuation length.

[0092] As an example, if the goal is to reduce the intensity of a physical phenomenon by about an order of magnitude, only a material with a thickness of about three times the attenuation length is needed as a shielding component, and the thicker the better.

[0093] The following can be considered for typical materials used for electromagnetic and radiation shielding, as well as their typical thicknesses.

[0094] • Gamma rays: Tungsten, several millimeters or more

[0095] X-rays: Lead, several centimeters or more

[0096] • UVA, B, C: Glass or polycarbonate, several millimeters to tens of millimeters

[0097] • Infrared (NIR) and infrared (FIR): Organic polyethylene, plastic film, ranging from a few millimeters to hundreds of millimeters.

[0098] Terahertz light: Metals (aluminum, copper, etc.), ranging from several millimeters to hundreds of micrometers.

[0099] Microwave: Metals (aluminum, copper, etc.), from a few millimeters to tens of millimeters

[0100] (Examples of shock-absorbing components and vibration-absorbing components)

[0101] For vibration resistance, it is effective to use non-elastic soft materials (gels or rubbers) to follow the vibration of functional materials. Alternatively, when the vibration frequency to be avoided is known, it can be achieved by joining materials with non-resonant lengths together.

[0102] Regarding thickness, common sense dictates that as the thickness increases, vibrations and impacts tend to be absorbed more easily.

[0103] Furthermore, for impact resistance, the ability to repel external forces through elasticity or tension is important, and this trend can be addressed by utilizing hard nonmetallic materials such as ceramics or the ductility of metals.

[0104] (Example of heat-resistant components)

[0105] For heat-resistant components, ceramics should be considered.

[0106] Effective heat resistance can be achieved by sandwiching an air layer with low thermal conductivity inside the shell using heat-resistant components. The thermal conductivity of the air layer can be reduced by lowering the pressure to near a vacuum.

[0107] Similar to the principle of a thermos, by setting up a vacuum layer, it is possible to prevent unexpected energy from the outside from reaching the inside.

[0108] Since the heat resistance is mainly provided by the depressurized air layer, it is desirable to thicken this part, but there is a trade-off: thickening this part will amplify the point diffusion or diffraction effect of the incident light from hole 3a.

[0109] Because air flows easily and exchanges heat with the outside as it flows, resulting in heat loss and impairing heat resistance, the air is kept inside, and the shielding wall uses solid materials such as ceramics with low thermal conductivity. Although the ceramic material itself also needs thickness from a heat resistance perspective, when the air layer is fully effective, it only needs to have the strength required to maintain the shape of the ceramic material itself (i.e., maintain the shape of the shell). The same applies to impact-absorbing components.

[0110] (Examples of corrosion-inhibiting components and bio-repellent components)

[0111] When the cause of the problem to be solved is known, materials corresponding to that cause should be used.

[0112] (Example of a heat dissipation component)

[0113] For heat dissipation components, materials that effectively release internal heat to the outside to maintain the internal temperature are used.

[0114] When only heat needs to be released to the outside, it is sufficient to either directly diffuse the heat to the outside or construct the outer wall using a material with high thermal conductivity. However, since heat dissipation is also required to maintain a constant temperature, once the heat is transferred to a material with a large heat capacity, even the same amount of heat dissipation can keep the internal temperature at a lower level.

[0115] For example, such as Figure 6 As shown in the cross-sectional view, the following configuration can be considered: a highly thermally conductive transparent component 5 is filled inside the housing formed by the protective component 3, which serves as a heat dissipation component.

[0116] To increase heat capacity, materials with high specific heat are preferred. For example, although air and water have high specific heat, metals, especially copper or aluminum, can be considered due to their low thermal conductivity.

[0117] Heat is effectively diffused within the component, and as the temperature rises, heat dissipation is effectively performed from the contact surface with the outside.

[0118] (Regarding heat resistance and heat dissipation)

[0119] Here, both heat resistance and heat dissipation functions aim to maintain the internal temperature, but their configurations differ depending on whether the heat source is external or internal. Furthermore, regardless of whether the heat source is internal or external, the configurations may also differ depending on whether the heat source is temporary or permanent.

[0120] (1) When the heat source is internal and permanent

[0121] This applies to situations where sensors and computing circuits are constantly running (such as environmental setup cameras).

[0122] • To minimize the temperature difference with the outside environment, heat dissipation is improved by covering the material with high heat capacity and thermal conductivity.

[0123] (2) When the heat source is internal and temporary

[0124] This applies to examples such as computation functions whose state changes due to some trigger (moving bodies, cameras, etc.).

[0125] Depending on the degree of temporality, in cases of short duration, thermal conductivity should be given priority in order to quickly dissipate heat to the outside.

[0126] (3) When the heat source is external and temporary

[0127] This applies to examples where the state changes due to external triggers (such as a moving object carrying a camera).

[0128] • To improve the heat storage capacity of the outer shell, materials with a large heat capacity are used, and to slow down heat diffusion into the interior, materials with high thermal conductivity are used, thereby improving the heat resistance.

[0129] (4) When the heat source is external and permanent

[0130] This applies to situations where the heat source is placed outdoors (such as an environmental setup camera).

[0131] In addition to heat resistance, it also needs to dissipate heat.

[0132] The following table shows the property information of typical materials used for heat resistance and heat dissipation.

[0133] [Table 1]

[0134] For thermal buffer materials to possess both heat resistance and heat dissipation properties, a high heat capacity is desirable. However, since heat capacity is a ratio that can be maintained relative to weight, volumetric specific heat is important for calculating the heat-to-size ratio. Column D in the table summarizes the volumetric specific heat, and the heat capacity can be obtained by multiplying this value by the volume.

[0135] On the other hand, regarding heat resistance and heat dissipation, for heat resistance, a low thermal conductivity is desirable to prevent external heat from being conducted to the interior, while the opposite is true for heat dissipation. Therefore, column E in the table shows the calculation results of volumetric specific heat × thermal conductivity, and column F shows the calculation results of volumetric specific heat / thermal conductivity.

[0136] It can be seen that materials with larger values ​​in column E have better heat dissipation performance, while materials with higher rankings in column F have better heat resistance.

[0137] When only heat dissipation is required, it is preferable to use materials that rank higher in heat dissipation performance, and diamond or copper are preferred. However, since the high transmittance of diamond renders the through-holes functionally meaningless, copper is sufficient.

[0138] However, since the light-transmitting portion 31 is closest to the heat source, it is important to dissipate heat quickly, and a material with high thermal conductivity is required. In this case, silver can be considered for the light-transmitting portion 31, for example. Furthermore, to enhance heat diffusion between the light-transmitting portion 31 and the imaging element 2, a transparent material with high thermal conductivity is used. Diamond is an example of such a material.

[0139] Note that, as a selection criterion for the light-transmitting part 31, it is important to choose a material that is not prone to thermal expansion, so that the hole 3a will not deform due to thermal expansion. In this case, since metals are prone to thermal expansion, silicon carbide or similar materials can be considered.

[0140] As can be seen from the above, the aperture 3a is preferably formed of silicon carbide, which is not easily thermally expanded and has relatively high thermal conductivity. The portion other than the light-passing portion 31 is formed by using a thicker copper material with both high volumetric specific heat and high heat transfer capacity to form a heat sink with increased heat capacity. The space between the imaging surface and the surface of the light-passing portion 31 is formed of a material with high thermal conductivity and high transmittance (e.g., diamond).

[0141] Here, considering the heat resistance function, we can consider using materials with general heat resistance function, that is, materials ranked higher in column F.

[0142] In addition, in order to maintain the internal temperature, heat dissipation is required to match the inflow rate, so materials with both heat resistance and heat dissipation functions are needed.

[0143] (Example of a hydrophobic component)

[0144] Regarding the protective component 3, which is a hydrophobic component, for example... Figure 7 As shown in the cross-sectional view, a component with a hydrophobic layer 36 formed on its surface can be considered. The hydrophobic layer 36 can be formed by coating to create a layer consisting of fine protrusions.

[0145] The hydrophobic layer 36 is configured to amplify the surface tension of the adhering droplets, and because the droplets maintain their shape due to surface tension and cannot adhere to the surface, an effect of jumping out to the outside while moving irregularly can be achieved (hydrophobicity). One of the features is that, since tiny dirt particles on the surface are entangled during this irregular movement, a dirt-cleaning effect can be achieved (lotus effect).

[0146] The protective component 3, which serves as a hydrophobic component, can be manufactured, for example, by coating a micro-protrusion layer on an opaque substrate and then forming one or more holes 3a (through holes) using a laser, stamping die, or the like.

[0147] Note that the above Figure 7 and Figure 6 An example is shown in which multiple holes 3a are formed in the light-passing part 31, but a similar configuration can also be used when the hole 3a is a single hole.

[0148] [1-3. About Image Correction]

[0149] As described above, when using a single pinhole camera with aperture 3a, blurring occurs in the captured image.

[0150] Moreover, in the configuration using multiple aperture 3a (hereinafter referred to as "multi-aperture camera"), the degree of blurring of the captured image becomes greater.

[0151] In view of this, the imaging device 1 is configured to correct the blur generated in the captured image when the imaging method using a pinhole camera or a multi-pinhole camera is used as a lensless imaging method.

[0152] Figure 8 This is a block diagram illustrating an example of the internal configuration of an imaging device 1 including such a blur correction configuration.

[0153] As shown in the figure, light from the subject shines onto the imaging surface of the imaging element 2 through the hole 3a of the light transmission part 31.

[0154] Although not shown in the figure, imaging element 2 has a pixel array section in which multiple pixels performing photoelectric conversion are arranged in two dimensions, and the electrical signal obtained for each pixel is converted by A / D conversion to obtain captured image data. For imaging element 2, for example, a charge-coupled device (CCD) type image sensor or a complementary metal-oxide-semiconductor (CMOS) type image sensor is used.

[0155] It is important to note that, in Figure 8 In this case, the number of holes 3a formed in the light-passing part 31 is one, but as described above, it is only necessary to form one or more holes 3a in the light-passing part 31.

[0156] In the imaging device 1, the captured image obtained by the imaging element 2 is input to the image correction unit 10.

[0157] The image correction unit 10 corrects the blurring in the captured image input from the imaging element 2 caused by the lensless imaging method. Thus, the corrected captured image is obtained as shown in the figure.

[0158] Here, in the case of using a pinhole camera with a single aperture 3a, relatively small blur is produced in the captured image. In this case, it is conceivable that the image correction unit 10 performs image sharpening processing as blur correction processing (processing to improve sharpness).

[0159] When using a multi-pinhole method with multiple holes 3a, the blur produced in the captured image is caused by the superposition of positional offsets of multiple images, as described below, and is therefore difficult to properly correct through conventional sharpening processes.

[0160] Figure 9 This is an explanatory diagram about the superposition of position offsets.

[0161] like Figure 9 As shown, when multiple apertures 3a are formed, multiple subject images are formed on the imaging surface by light passing through each aperture 3a. The imaging positions of these multiple subject images on the imaging surface vary depending on the position of the aperture 3a through which the light passes, and therefore the imaging element 2 receives light from the superimposed image of the multiple subject images with offset positions. Thus, the captured image is obtained as a blurred image with blurred edges of the subject.

[0162] When using a multi-pinhole method, the image correction unit 10 corrects the blur caused by the superposition of positional offsets as described above (details will be described later).

[0163] Here, in the case of the multi-pinhole method, the blurring pattern is determined by the arrangement pattern of the pinholes 3a, and therefore the blurring can be corrected by utilizing the image transformation of this arrangement pattern, i.e., restoring the unblurred image. Since this restoration utilizes the fitting of the arrangement pattern within the blurred image, it is desirable to ensure that as few similar partial patterns as possible are generated within the pattern (so that the autocorrelation is reduced).

[0164] This pattern can be achieved, for example, by arranging micropores 3a like a QR code (a registered trademark).

[0165] Figure 10 An example of the pattern forming this hole 3a is shown.

[0166] In the figure, the black dot indicates the location where hole 3a is formed.

[0167] Figure 10 The pattern of the hole 3a shown is as follows: For multiple unit regions obtained by dividing a rectangular region of size A×A into rectangular regions of size m×m, a rectangular hole 3a of size b×b is set in a unit region selected based on a predetermined rule including randomness.

[0168] Figure 11 Is it being adopted in Figure 10 An illustrative diagram of a blurred image obtained in the case of pattern formation shown, wherein Figure 11 A shows the original image and Figure 11 B shows a blurred image.

[0169] Regarding Figure 10 The pattern for forming the aperture 3a shown is constructed using the sensor distance s (i.e., the distance between aperture 3a and the imaging surface), the pixel pitch Pt of the imaging element 2, the wavelength λ of visible light, etc., with parameters m and b defined as α = λs / πb, Pt = 4α, and m = b + 4α. In these expressions, the expression for α corresponds to the diffraction width (one-sided) derived from the diffraction of parallel light generated by the rectangular aperture 3a on the imaging surface. The expression for Pt represents the condition that the diffraction width with margin (one-sided) matches the resolution of the pixels of the imaging element 2. When Pt and λ are 2 μm and 0.4 μm respectively, it can be seen that the ratio of the size of aperture 3a to the sensor distance s is preferably approximately 1:25. When s is 1000 μm, b is preferably approximately 40 μm. The expression for m is used to prevent interference (crosstalk) between adjacent apertures 3a, and therefore m is approximately 42 μm.

[0170] Note that this design approach aims to maximize the efficient use of the imaging element 2, but depending on the shape (including thickness) and strength of the protective component 3 and the manufacturing method of the hole 3a, this size may not be achievable, and therefore appropriate adjustments are expected.

[0171] Furthermore, since several pattern examples can be considered, a pattern that can match the structure of the protective component 3 while maintaining image restoration performance is used in several pattern examples.

[0172] For example, although a grid shape is desirable for electromagnetic shielding, the QR code pattern is constrained by the boundary preservation between holes 3a, and thus grid-like shielding can be directly achieved, which is efficient (the spacing of the wire mesh does not need to be consistent with the spacing of the code).

[0173] Other well-known patterns include Fresnel zone plates, etc.

[0174] Examples that could be considered include the use of geometric patterns such as Voronoi patterns and Delaunay triangles. Additionally, graphics with elements serving as identifiers, such as those obtained by transforming grid patterns (e.g., the design for the Tokyo 2020 Olympics), could also be used.

[0175] Note that as the number of holes 3a increases, forming each hole 3a one by one with a laser takes a lot of manufacturing time. To form holes 3a simultaneously, it is possible to use multiple lasers, but if it is desired to produce a large number of identical patterns, a stamping die can be produced and the holes 3a as through holes can be stamped out.

[0176] Here, when multiple apertures 3a are formed, since the image of the subject is not formed on the imaging surface, the correction process of the image correction unit 10 can be restated as a process that performs an image conversion equivalent to imaging.

[0177] This blur correction processing, comparable to image transformation, can include cases using image transformation functions derived from physical models (i.e., implemented as rule-based processing) and cases using image transformation functions derived from learning (using trained artificial intelligence (AI) models). The latter, derived from learning, consists of functions whose function structure contains parameters that may not have explicit physical meaning, distinguished from p-information, and whose values ​​are determined through statistical learning from data. For both physically derived and learning-derived functions, the function structure is typically contained within the computation module, and the parameters are selected and loaded from external resources.

[0178] The physical model is based on the following description: where the image projected onto the imaging surface from multiple apertures 3a can be considered as each point of the subject as a point light source, and the shadows of the apertures 3a are superimposed on the image on the imaging surface, as described above. Figure 9 As described in [the document], and can be represented by a convolution function.

[0179] In the case derived from the physical model, the blur correction process, which is an image transformation, only requires the computation of deconvolution as its inverse process, and the image transformation function in this case represents the deconvolution function for restoring the original image from the captured image. Furthermore, since it is known that the convolution function can be implemented using the Fast Fourier Transform algorithm, the implementation is formulated using Fourier Transform and Inverse Fourier Transform.

[0180] When the shape of aperture 3a and the precise values ​​of the parameters of the optical system are known, the image transformation function derived from the physical model can perform accurate image reproduction, but there are problems such as the need for stable behavior when information cannot be captured by the physical model, or when errors still exist even if it can be captured.

[0181] For example, in the case of an ideal pinhole camera, the image transformation function can be easily derived because there are few parameters. However, as the number of holes 3a increases, the positional relationships become more complex, and factors such as the thickness of the holes 3a make it difficult to derive the function easily.

[0182] When using multiple holes 3a, the difficulty in deriving functions may increase the burden of development work.

[0183] Therefore, it is worth considering using image transformation functions derived from learning. Specifically, using knowledge from deep learning is effective.

[0184] Reference Figure 12 Examples describing specific learning methods.

[0185] Figure 12 A shows an example of a method for generating a learning dataset for learning, and Figure 12 B shows an example of machine learning (deep learning) using the generated learning dataset.

[0186] To generate a learning dataset, use, for example Figure 12 The imaging device 1' shown in Figure A is used to generate learning data. The imaging device 1' for generating learning data only needs to be configured to obtain a blurred image caused by the aforementioned positional offset as the captured image, and the example shown uses an image obtained from the aforementioned... Figure 8 The image correction unit 10 is omitted in the imaging device 1, but the imaging device 1 can also be used when it is able to output a blurred image before correction.

[0187] The learning data is generated by sequentially displaying various types of image data obtained by capturing scenes to be used as learning targets on the display 20, and capturing each displayed image using the imaging device 1' used to generate the learning data.

[0188] In this case, machine learning is performed in the form of supervised learning, such as... Figure 12 As shown in B, the blurred image captured by the imaging device 1' used to generate learning data is provided as learning input data to the learner 21 used to generate an AI model that performs blur correction, and the image data of the captured object, which is the blurred image, is used as teacher data.

[0189] Therefore, it is possible to create AI models that can restore the original image from a blurred image (i.e., achieve blur correction).

[0190] Alternatively, during learning, a common subject can be captured using a regular camera with a lens and an imaging device 1' for generating learning data. The image captured by the camera with the lens can be used as teacher data, and the blurred image captured by the imaging device 1' for generating learning data can be used as learning input data.

[0191] When the depth of the subject is sufficiently small relative to the distance from hole 3a to the subject (e.g., less than 10%), it can be as follows: Figure 12 The example replaces the photographed object with an image displayed on the monitor 20.

[0192] Here, learning in this case can be achieved by: preparing a parameterized function in advance to convert the captured image to the final output image, and optimizing the parameters by taking the captured image as input and making the output image approximate the subject image (to minimize the squared brightness error).

[0193] At this point, for optimization, iterative processing of the error backpropagation method is performed, and statistical update processing is used to update the iteration error. One of the characteristics of deep learning is that the algorithm has been well developed, and complex functions can be learned by selecting an appropriate algorithm from standard algorithms (ADAM-W, etc.) and adjusting parameters such as the learning rate.

[0194] Another characteristic of deep learning is that various complex functions are modeled by parametric composition of composite functions through simple function sets (multidimensional inputs, multidimensional outputs) (lightweight, approximation).

[0195] The simple set of functions is initially mathematical functions, such as linear functions with weighted parameters, convolutional functions, and nonlinear activation functions. However, attention mechanisms, gating mechanisms, and other modules that combine these functions, such as transformer modules, gated multilayer perceptrons, and Long Short-Term Memory (LSTM) modules, are sometimes also considered functions. Attention mechanisms weight data based on the relevance of each region in multidimensional data, while gating mechanisms control the flow of information by weighting the input and output of a function according to the importance of the state. These mechanisms were initially applied to sequential data such as speech, but can also be applied to two-dimensional data such as images by converting the data into a one-dimensional sequence. When the input data is used for images, its characteristic compared to convolutional functions is its ability to efficiently capture information from distant pixels.

[0196] In image conversion, it is common to use a structure called an autoencoder, which combines an encoder mechanism that extracts features from an image and a decoder mechanism that reconstructs the image from the features.

[0197] The encoder section is responsible for separating the arrangement of aperture 3a and subject pixels from the captured image. However, since the diffusion region in the captured image constituting a single subject pixel involves a wide range, it requires efficient utilization of information from distant pixels. Previously, weighted convolutional functions (CNNs) were indispensable in image processing, but due to their low efficiency in utilizing the correlation of information from distant pixels, modules that directly connect one or more blocks of image transformers (ViTs) or gated multilayer perceptrons (GMLPs) are used. The decoder section can be formed by stacking several stages of blocks including ordinary multilayer perceptrons or convolutional functions (small kernel size) with minimal peripheral influence.

[0198] The parameters include the formation pattern information of hole 3a and the image conversion information. However, according to the learning method, since the formation pattern information of hole 3a is embedded during the learning process, only the image conversion information can be a parameter.

[0199] Figure 13 A model is shown that uses a gated multilayer perceptron (GMLP) as the encoder and a fully connected neural network (FNN) as the decoder for the image transformation function.

[0200] Encoders and decoders are mathematically synthetic transformation functions of tensors (numerical arrays) with parameters. A large number of parameters that do not have direct physical meaning, obtained through data learning, are stored and provided when the function is executed.

[0201] [1-4. Solutions to problems caused by hole thickness]

[0202] In a lensless camera (such as a pinhole camera), it is ideal for the thickness of the hole 3a to be negligible. However, in cases such as imaging device 1 where the hole 3a needs to be formed in a protective component 3 with a protective function, the thickness may not be negligible, such as when the degree of protection depends on the thickness of the protective component 3 or when a supporting substrate is required.

[0203] When the hole 3a of the pinhole camera has a thickness, the micro-through hole takes the shape of a column and the following phenomenon occurs: light from the subject area with a large angle of incidence far from the optical axis of the camera is completely absorbed or deflected by irregular reflection within the hole 3a, and even in the area that is not deflected, the transmittance decreases towards the outside.

[0204] Figure 14 The schematic diagram illustrates the path of light emitted from a specific point p on the subject when the aperture 3a has thickness. The solid arrows show the path of the straight-line light that travels without reflection within the aperture 3a (not reflected at the interface between the aperture 3a and the protective member 3), while the dashed arrows show the path of the reflected light that is reflected within the aperture 3a.

[0205] As a trend, it can be seen that the greater the thickness of aperture 3a, the easier it is for reflection to occur in the form of multiple reflections. In addition, although it depends on the material of the protective component 3, light other than directly incident light may be absorbed within aperture 3a.

[0206] Problems caused by the thickness of hole 3a can be listed as follows (a) to (e).

[0207] (a) Increase in image noise due to irregular reflections within aperture 3a

[0208] (b) Decrease in light transmittance due to absorption within aperture 3a

[0209] (c) Transmission vanishing depends on the angle of incidence of the light on the subject.

[0210] (d) Learning difficulties caused by the loss of information about the periphery of the subject; (e) Reduced operating efficiency of imaging element 2 due to loss of transmission As a countermeasure to the problem in (a) above, such as Figure 15 As shown in the cross-sectional view, it is possible to form a reflection suppression component 37 in the region surrounding the hole 3a of the protective component 3. For example, it is possible to apply an anti-reflection (AR) coating to the region surrounding the hole 3a in the protective component 3. In this case, a light-absorbing material can be used as the reflection suppression component 37, which absorbs the target wavelength (such as visible light) and converts it into heat energy or an infrared wavelength other than the target wavelength.

[0211] Alternatively, if the protective component 3 is formed by using metal, it is also possible to form an oxide film at the interface with the hole 3a to eliminate gloss (i.e., to give a reflection suppression function).

[0212] Note that in Figure 15 In this case, the reflection suppression component 37 is formed in only one hole 3a, but when multiple holes 3a are used, the reflection suppression component 37 can be formed for each hole 3a.

[0213] Here, when performing blur correction using the image conversion function derived from the physical model described above, it is appropriate to form the reflection suppression component 37 as described above. This is because the reflection component within the aperture 3a becomes negligible when deriving the image conversion function, and the burden of the deriving work can be reduced.

[0214] In the case of fuzz correction via an AI model, when the reflection pattern within the aperture 3a has reproducible regularity, it is learned and absorbed, and therefore the measure of forming a reflection suppression component 37 is not necessary.

[0215] Furthermore, as a countermeasure to (b) and (c), transmittance correction could be considered in the image conversion used for blur correction.

[0216] Since the shape and size of aperture 3a and the sensor distance s are known, the transmittance of the subject's light can be predicted when the distance between the subject and aperture 3a is known. In the image conversion from the captured image to the recovered image, only corrections through the inverse transform of this transmission process need to be added.

[0217] Note that the expression used for correction will be described later.

[0218] When using an AI model to perform image transformation as a blur correction, transmittance correction as described above is not necessarily required, since the transmittance correction element is also included in the AI ​​model.

[0219] As a countermeasure to (d), measures such as removing or masking the outer edge of the teacher image used for learning, or enlarging the teacher image used for learning and then cropping it back to its original size, can be considered.

[0220] When the image displayed on the display 20 is used to generate learning data, since the distance between the subject and the aperture 3a is known, it is possible to predict the area that will become irretrievable due to the absorption and scattering of light from the subject.

[0221] Learning efficiency can be improved by removing or masking teacher images in areas that cannot be restored.

[0222] Here, the problem with (d) mainly needs to be considered when the distance from the subject is uncertain, and the area to be covered will be described later.

[0223] It should be noted that areas that cannot be restored can be areas where the transmittance drops below a predetermined level even if light does not fully reach the imaging surface.

[0224] As a countermeasure to (e), the sensor distance s, which is the distance between the aperture 3a and the imaging surface, can be set to a distance such that the size of the subject illumination range on the imaging surface side is substantially the same as the size of the imaging range of the imaging element 2.

[0225] Reference Figure 16 Describe the range of light illuminating the subject.

[0226] As shown in the figure, the subject light illumination range is the range of subject light that is illuminated to the imaging surface side through all holes 3a.

[0227] The imaging range of imaging element 2 refers to the range of pixels formed for imaging in the pixel array section (pixel forming area) of imaging element 2.

[0228] refer to Figure 16 It is understandable that when the sensor distance s is short, the size of the subject's illumination range becomes smaller relative to the imaging range, and the imaging range cannot be utilized effectively. Therefore, by setting the sensor distance s so that the size of the subject's illumination range is substantially the same as the size of the imaging range, the operating efficiency of the imaging element 2 can be improved.

[0229] Here, "substantially consistent" is not limited to strict consistency, but rather has a certain range, such as the range of errors that may actually occur relative to the design value, or the range that will produce substantially similar effects.

[0230] Reference Figure 16 Describe the specific conditions used to make the size of the illuminated area of ​​the subject consistent with the size of the image area.

[0231] As shown in the figure, when the size of the subject is D×D, the size of the light-passing part 31 (the size of the light-passing area) is a×a, the size of the aperture 3a is b×b, the size of the imaging range is c×c, the distance between the aperture 3a and the imaging surface (the distance from the subject side surface of the light-passing part 31 to the imaging surface) is the sensor distance s, and the thickness of the aperture 3a is the thickness t. The condition under which the size of the subject light illumination range and the size of the imaging range are consistent is as follows.

[0232] ca : st = 2b : t

[0233] Figure 17 It is an explanatory diagram of the path of the light rays emitted from the outermost peripheral aperture 3a of the subject at the outermost periphery, and specifically shows the path of the light rays that travel outward without being deflected in the light rays that have entered the outermost peripheral aperture 3a of the subject at the outermost periphery.

[0234] from Figure 17 It can be seen that the settings used to ensure the size of the subject's illumination range and imaging range are consistent with each other are independent of the subject's position (and). Figure 16 The distance r of the subject shown is determined independently.

[0235] Typically, since the shape of the hardware device is determined during use, a, b, c, and t are fixed values, and the efficient utilization of the imaging element 2 can be achieved by specifying the sensor distance s as follows.

[0236] [Mathematical Expression 1] ... [Expression 1] Here, in the implementation, it has been mentioned that when using an AI model to perform blur correction on multiple holes 3a, a display 20 is used during the learning of the AI ​​model. However, when the distance from the holes 3a to the display 20 is inappropriate, the imaging device 1' used to generate learning data cannot capture the entire area of ​​the image displayed on the display 20, cannot generate appropriate learning input data, and there is a possibility that the learning efficiency may be reduced.

[0237] Therefore, when generating learning data, it is only necessary to determine the distance between hole 3a and display 20 (corresponding to...). Figure 16 The distance r of the subject in the image, thus making Figure 16 The size of the imageable area shown is substantially the same as the size of the display surface of the display 20.

[0238] exist Figure 16 The conditions under which the size of the imageable area and the size (D×D) of the display surface of the display 20 are consistent are as follows.

[0239] da : r = 2b : t

[0240] Figure 18 This is an illustration of the path of light rays that can pass through the outermost periphery of the subject through the outermost aperture.

[0241] from Figure 18 It can be seen that the configuration that makes the size of the imageable area consistent with the size of the display surface of the display 20, and the distance between the aperture 3a and the imaging surface (and) Figure 16 The sensor distance s shown is determined independently.

[0242] Typically, since the shape of the hardware device is determined during use, a, b, c, and t are fixed values, and the distance between the specified hole 3a and the display surface of the display 20, i.e., the subject distance r (e.g., ... Figure 16 As shown, the distance from the subject-side surface of the light-passing section 31 to the subject is as follows, which allows the imaging device 1' used to generate learning data to capture the entire area of ​​the image displayed on the display 20.

[0243] [Mathematical Expression 2] ... [Expression 2] <2. Second Implementation Method> Next, the second implementation method will be described.

[0244] In the first embodiment, the description was based on the premise that the size and distance of the subject are fixed. However, in actual imaging, the size and distance of the subject are uncertain and may vary. In other words, it may be necessary to adapt to subjects of different sizes and distances.

[0245] However, when using reference Figure 12 In the case of the learning method described, since the AI ​​model only learns at the subject distance r (the distance between the hole 3a and the display surface of the display 20) set when generating the learning data, the blur correction performance will decrease with changes in the subject distance r.

[0246] Therefore, in the second embodiment, the following method is adopted: the AI ​​model used by the image correction unit 10 switches between multiple AI models derived for each distance r of different subjects.

[0247] Figure 19 This is a block diagram showing an example of the internal configuration of the imaging device 1A as a second embodiment.

[0248] In addition, in the following description, the same reference numerals are used for parts that are the same as those already described, and their descriptions are omitted.

[0249] and Figure 8 The difference between the imaging device 1 of the first embodiment shown is that a control unit 11, a memory unit 12 and an operation unit 13 are added.

[0250] The control unit 11 includes a microcomputer having, for example, a central processing unit (CPU), a read-only memory (ROM), and a random access memory (RAM), and performs overall control of the imaging device 1A by the CPU performing various types of processing according to the program stored in the ROM.

[0251] The memory unit 12 is configured as a storage device such as a solid-state drive (SSD) or a hard disk drive (HDD) and stores various types of information. Specifically, the memory unit 12 of this embodiment stores multiple AI models as candidates for AI models used for blur correction in the image correction unit 10. These multiple AI models are AI models obtained by learning for each distance r of different subject distances. Specifically, the multiple AI models are obtained by... Figure 12 In the learning of the AI ​​model described herein, multiple AI models are derived by performing multiple learning operations at different arrangement distances of the display 20 relative to the imaging device 1' used to generate learning data.

[0252] Here, as an example, the arrangement distance of the imaging device 1' used to generate learning data relative to the display 20 is learned at each of three distances (i.e., a first distance, a second distance, and a third distance), and the storage unit 12 stores a first AI model M1 as an AI model learned at the first distance, a second AI model M2 as an AI model learned at the second distance, and a third AI model M3 as an AI model learned at the third distance.

[0253] The operation unit 13 comprehensively represents various operators such as buttons, keys, and touch panels formed on the imaging device 1A. User operations are detected by the operation unit 13, and signals corresponding to input operations are interpreted by the control unit 11.

[0254] The control unit 11 controls the switching between multiple AI models stored in the storage unit 12 using the AI ​​model used by the image correction unit 10.

[0255] In this example, it is assumed that there are three types of subject distances r in the imaging, namely near distance, medium distance and far distance, and memory unit 12 stores a first AI model M1, a second AI model M2 and a third AI model M3 as multiple AI models obtained by learning for each of these three types of distances.

[0256] Here, the user performs a specified operation at close range, medium range, or long range through the operation unit 13 according to the imaging scene, and the control unit 11 performs the process of selecting the AI ​​model corresponding to the specified operation from the AI ​​models stored in the storage unit 12 and setting it in the image correction unit 10.

[0257] Note that the switching control of the AI ​​model is not limited to being performed based on user operation as described above. For example, it could be performed in response to instructions from an external device. Alternatively, it could be considered to include a distance measurement sensor in the imaging device 1A and switch the AI ​​model based on the subject distance r measured by the distance measurement sensor.

[0258] Furthermore, although three distances (i.e., the first to the third distances) have been shown above as assumed subject distances r, the assumed subject distances r are not limited to three and can be multiple. By increasing the number of assumed subject distances r, the range of subject distances that can be accommodated can be expanded.

[0259] Here, although the above description is based on the premise of using an AI model to perform blur correction in the image correction unit 10, the correction algorithm can also be switched according to the subject distance r when performing blur correction through rule-based processing.

[0260] Here, a description of transmittance will be given.

[0261] Qualitatively, transmittance decreases towards the outer edge of the image and in some cases no transmission occurs, but transmittance relative to distance and coordinates on the subject is calculated as follows.

[0262] [Mathematical Expression 3] ... [Expression 3] Here, since the size and distance of the subject are large enough, it is assumed that the angle of incidence of any hole 3a of the light-passing part 31 from the same point on the subject is equal.

[0263] In the light emitted from point p on the subject, when considering the light quantity distribution of transmitted light incident on one end of a rectangular aperture 3a with side length b and exiting from the other end, the ratio of the area of ​​the transmitted light to the opening area of ​​aperture 3a is equivalent to the transmittance.

[0264] Figure 20 The area of ​​the transmitted light is schematically shown.

[0265] When there exists an irradiation area, represented by the area of ​​the transmitted light, due to the proportional relationship bu:t=p:r in the vertical dimension u1 and horizontal dimension u2 of the rectangle of the irradiation area, the point p of the object is obtained as (p1, p2), the distance r of the object, the rectangular width b and thickness t of the aperture 3a, u=b-pt / r, and the transmittance is 0 when br < pt. The transmittance is obtained from the area ratio as follows.

[0266] [Mathematical Expression 4] ... [Expression 4] Such transmittance is a quadratic function of the reciprocal of the subject distance *r*, and as the subject distance *r* increases, the transmittance approximates 1 when *br* is sufficiently small compared to *td*. That is, for sufficiently distant subjects, a general model function can be used, but since the learning characteristics change with decreasing distance, the model function can, for example, have roughly the same functional form. The difference then lies only in the parameters. For example, correction algorithms can be derived for distances where typical features might occur (i.e., distance *td / b*, distance 10 *td / b*, and distance 0.1 *td / b*), and at each execution, only the algorithm with the closest distance needs to be selected and used for blur correction.

[0267] Here, when learning is considered in the case of using an AI model for blur correction, the condition for the existence of a region with 0 transmittance on a display of size d×d is 2br < td, and at this time, outside the rectangular region with a center width of br / t on the display 20, light cannot reach the imaging surface and becomes irretrievable.

[0268] Since attempting to learn the restoration function for the unrecoverable region is inefficient, as mentioned above, learning efficiency can be improved by removing or masking the teacher image of the unrecoverable region. Alternatively, measures such as enlarging the correct image, cropping the unrecoverable region, and restoring it to its original size can also be considered.

[0269] <3. Third Implementation Method>

[0270] The third embodiment aims to absorb differences in the subject distance r by changing the thickness and size of the aperture 3a. Specifically, the thickness t of the aperture 3a is made thinner, or the size of the aperture 3a is made larger, because the distance to the subject is shorter, thus suppressing the missing parts of the image at the outer periphery of the subject.

[0271] In this embodiment, when changing the thickness and size of the aperture 3a, an aperture changing mechanism is used, which is a mechanism for changing the thickness or size of the aperture 3a used for imaging.

[0272] Figure 21 This is a diagram illustrating a rotary table-based hole-changing mechanism as an example of a hole-changing mechanism.

[0273] In this case, a plurality of light-passing portions 31 are formed on the generally disc-shaped turntable portion 40, and the plurality of light-passing portions have holes 3a with different thicknesses t and sizes.

[0274] Specifically, in Figure 21 In example A, three light-passing sections 31 with different thicknesses t are formed at different positions of the turntable section 40, specifically light-passing sections 31-1, 31-2, and 31-3.

[0275] Note that, also in the third embodiment, the number of holes 3a formed in the light-passing portion 31 can be single or multiple.

[0276] Figure 21 B shows a cross-sectional view of the imaging device 1B as a third embodiment, wherein the turntable 40 is configured to switch the light transmission section 31 located on the imaging element 2 by rotation.

[0277] Therefore, the thickness t of the aperture 3a used for imaging can be switched.

[0278] Figure 22 This is a block diagram showing an example of the internal configuration of the imaging device 1B as a third embodiment.

[0279] With the above Figure 19The difference in the imaging device 1A shown is that an actuator 15, such as a motor, is added as a drive source for rotating the drive turntable 40, and a control unit 11B is provided instead of the control unit 11. In this case, it is not necessary to switch the AI ​​model according to the distance to the subject, and since it is not necessary to store multiple AI models as switching candidates, the storage unit 12 is omitted.

[0280] The control unit 11B controls the switching of the light transmission unit 31 used for imaging.

[0281] Specifically, the control unit 11B controls the actuator 15 so that the light transmission section 31 corresponding to the distance specification operation performed by the user through the operation unit 13 is located on the imaging element 2.

[0282] Note that the switching control of the light transmission unit 31 used for imaging is not limited to being based on user operation as described above, and may also be performed in response to an instruction from an external device, or, if the imaging device 1B is equipped with a distance measurement sensor, the switching may be performed based on the subject distance r measured by the distance measurement sensor.

[0283] Furthermore, although three distances (i.e., the first to the third distances) have been shown above as being assumed to be the subject distance r, the assumed subject distance r is not limited to three types and can be multiple.

[0284] Alternatively, a configuration can be used that switches the size of hole 3a instead of the thickness t of hole 3a, or a configuration that switches both the thickness t and the size.

[0285] Alternatively, measures can be taken to prevent light that is not used for imaging from leaking into the interior of the imaging device 1B through the light-passing section 31. For example, a window layer can be formed on the lower side of the turntable section 40 to guide the transmitted light from the light-passing section 31 to the imaging element 2 side while preventing such light leakage.

[0286] Here, although the above description gives an example of switching the light transmission section 31 used for imaging by means of a turntable, by using the turntable method, it is possible to switch the thickness or size of the hole 3a used for imaging while ensuring the sealing of the protective component 3.

[0287] However, the use of a turntable is just one example, and for example, as... Figure 23 As shown, the configuration of the light-passing section 31 used for imaging can also be switched by a sliding method.

[0288] exist Figure 23In the example, a sliding portion 41 is formed in the imaging device 1B, which can slide along a direction parallel to the upper surface of the protective member 3 (i.e., the sliding direction Ds). In the sliding portion 41, a plurality of light-passing portions 31 with different thicknesses t or aperture 3a sizes are arranged in a direction parallel to the sliding direction (three examples of light-passing portions 31-1 to 31-3 are used here), and the light-passing portion 31 used for imaging is switched by the sliding drive of the sliding portion 41, that is, the thickness t or size of the aperture 3a used for imaging is switched.

[0289] Moreover, this sliding method allows for switching the thickness or size of the hole 3a used for imaging while ensuring the sealing of the protective component 3.

[0290] Here, through comparison Figure 23 and Figure 21 As can be seen from B, the use of a turntable reduces the gap generated on the protective component 3, which means it is suitable in terms of protection performance.

[0291] Note that in order to adapt to changes in the subject distance r, it is also possible to consider setting up an imaging device that corresponds to a different subject distance r, such as an imaging device forming a light-passing section 31-1, an imaging device forming a light-passing section 31-2, and an imaging device forming a light-passing section 31-2.

[0292] Here, it is also possible to simultaneously switch the thickness t or size of the hole 3a described in the third embodiment and switch the correction algorithm described in the second embodiment. For example, it is possible to switch the correction algorithm described in the second embodiment for a certain range of subject distance r, and switch the thickness t or size of the hole 3a described in the third embodiment for a range of more or less distant subject distance r.

[0293] <4. Applications of Imaging Devices>

[0294] Various applications of the imaging device described in the implementation method can be considered.

[0295] For example, it could be considered as a mobile camera mounted on a flying vehicle such as a drone or a vehicle such as a car.

[0296] Specifically, in addition to dashcams and blind spot assist cameras, surveillance cameras mounted on drones and inspection cameras for hard-to-access locations could be considered.

[0297] In any moving camera, the accumulation of residue caused by the evaporation of water droplets containing dirt is a problem.

[0298] Furthermore, shock absorption can be considered a problem specific to mobility. Since it is believed that damage will begin from the light-passing part when the light-passing part 31 alone cannot absorb the shock, countermeasures can also be considered for this situation.

[0299] As a protective function in this situation, hydrophobic functions that achieve natural cleaning through the lotus leaf effect and impact absorption functions utilizing the elasticity or extensibility of rubber, etc., can be considered. A structure could be adopted as follows: using an impact-absorbing material as the main material, coated with a hydrophobic coating, and finally providing a hole 3a (through hole).

[0300] In addition, various types of surveillance cameras can be considered (for people, animals, plants, and the natural environment (weather photos, satellite photos)).

[0301] Cameras designed for human use typically have features to prevent dirt buildup.

[0302] Cameras used for monitoring animals and plants are particularly well-suited for livestock and agriculture, but for smaller markets, they can also be considered for monitoring pets, gardening, and home vegetable gardens. Common dirt-accumulation-avoidance materials can also be used for these cameras.

[0303] For the natural environment, there is also the monitoring of river floods and ocean waves, but especially where cameras are set up in coastal areas, it is desirable to have a mechanism to prevent salt accumulation. To avoid salt damage, materials that inhibit crystal growth can be considered.

[0304] In natural environments such as high altitude, extreme heat, and polar regions, heat resistance and insulation are required. Therefore, a hydrophobic layer and through holes are set around a heat-resistant material.

[0305] For capturing meteorological and satellite images, radiation-resistant shielding materials are used to prevent damage to internal circuitry from cosmic heavy particle beams. Hydrophobic treatment can be omitted.

[0306] In addition, applications in special environments (high-voltage power line inspection, nuclear power plant accidents) can also be considered.

[0307] For the inspection of high-voltage transmission lines, electromagnetic shielding materials can be considered.

[0308] For nuclear power plant monitoring and accident investigation, the use of radiation shielding materials can be considered.

[0309] <5. Variations>

[0310] Note that the implementation methods are not limited to the specific examples described above, and can be configured in various variations.

[0311] For example, although the above description illustrates forming the entire housing of the imaging device as the protective component 3, the protective component 3 only needs to be formed at a location that at least covers the imaging surface of the imaging element 2.

[0312] Furthermore, although the above description is based on the premise that the opening shape of hole 3a is square, the opening shape of hole 3a can also be other shapes such as circles.

[0313] In addition, the use of measuring components to replace protective components 3 can be considered.

[0314] For example, measuring materials (thermocouples, piezoelectric elements) for inspecting the internal environment of the imaging device can be considered. Specifically, by placing thermocouples or the like near the light-passing part 31, measuring internal heat, and operating a heat protection circuit, internal protection can be achieved.

[0315] Alternatively, piezoelectric elements can be arranged.

[0316] In this case, the thickness of the light-passing portion 31 (hole 3a) is determined by the thickness used as a thermocouple or piezoelectric element.

[0317] <6. Summary of Implementation Methods>

[0318] As described above, the imaging apparatus (1, 1A or 1B) as an embodiment includes: an imaging element (2) for acquiring a captured image; and a protective member (3) for at least covering the imaging surface of the imaging element to protect the imaging element, and is configured to guide light from the outside to the imaging surface through a hole (3a) extending in the thickness direction.

[0319] According to the above configuration, it is possible to obtain captured images through a mechanism similar to that of a pinhole camera, while at least protecting the imaging element inside the device.

[0320] Therefore, according to this embodiment, it is possible to appropriately balance imaging function and protection function for internal components in an imaging device.

[0321] Furthermore, in the imaging apparatus implemented as an example, the protective component is a shielding component that shields the interior of the apparatus from the outside world, or a discharge and circulation component that has an external discharge or internal circulation function to improve the internal environment of the apparatus.

[0322] Examples of shielding components include electromagnetic shielding components, radiation shielding components, shock absorption components, vibration absorption components, heat-resistant components, corrosion-inhibiting components, and biological repellent components, and examples of discharge and circulation components include heat dissipation components, ventilation components, humidification components, and drying components.

[0323] By using these shielding components, discharge components, and circulation components as protective components, the internal components of the device can be appropriately protected according to the assumed use cases.

[0324] Furthermore, in the imaging device implemented as an example, a reflection suppression component (37) is formed in the area surrounding the hole in the protective component.

[0325] Therefore, the reflection of incident light at the aperture interface is suppressed.

[0326] Therefore, noise in the captured image caused by reflected light generated by the aperture can be reduced.

[0327] Furthermore, in the imaging device implemented as an example, multiple holes are formed.

[0328] By having multiple apertures, the amount of incident light on the imaging surface is increased.

[0329] Therefore, it is easy to adapt to imaging in dark places and high-speed imaging.

[0330] Furthermore, the imaging apparatus of this embodiment includes an image correction unit (10) that corrects the blurring in the captured image caused by the superposition of the position offset of the images of each hole on the imaging surface.

[0331] Therefore, even when using multiple apertures, it is possible to obtain a properly captured image that corrects for blur caused by positional offset superposition.

[0332] Furthermore, in the imaging apparatus implemented as an example, the image correction unit uses an AI model to correct blur.

[0333] Therefore, it is no longer necessary to create an inverse function for achieving fuzz correction through rule-based processing.

[0334] Therefore, the burden of development work in implementing fuzzy correction can be reduced.

[0335] Furthermore, in the imaging device (1) as an embodiment, the sensor distance, which is the distance between the aperture and the imaging surface, is set to the distance between the range of subject light illuminating the imaging surface side through all the apertures, i.e., the size of the subject light illumination range, which is substantially the same as the size of the imaging range of the imaging element.

[0336] Therefore, the imaging efficiency of the imaging element is maximized.

[0337] Furthermore, the imaging apparatus as an embodiment includes: an image correction unit (10) that corrects blur in the captured image due to lensless imaging; and a control unit (11) that controls the correction algorithm in the image correction unit to switch between multiple correction algorithms derived for each distance for different subject distances.

[0338] Therefore, it is possible to expand the range of subject distances from which appropriate blur correction can be performed.

[0339] Therefore, even when the imaging device is used in applications that assume multiple subject distances (e.g., close-range subject imaging, medium-range subject imaging, and long-range subject imaging), the correction algorithm can be switched to appropriately adapt to each application. This improves the functionality of the imaging device by expanding the range of adaptable subject distances.

[0340] Furthermore, in the imaging device (1A) implemented as an example, the image correction unit uses an AI model to correct blur, and the control unit controls the AI ​​model used by the image correction unit to switch between multiple AI models derived for each distance for different subject distances.

[0341] Therefore, in cases where AI models are used for blur correction, the range of subject distances that can be appropriately blurred can be expanded, and the functionality of the imaging device can be improved.

[0342] Furthermore, the imaging apparatus (1B) as an embodiment includes an aperture changing mechanism (turntable part 40 or sliding part 41), which is a mechanism for changing the thickness or size of the aperture used for imaging.

[0343] By changing the thickness or size of the aperture, the size of the imageable area and the range of light illumination on the subject can be altered.

[0344] Therefore, it is possible to adjust the imaging angle. In addition, it is possible to expand the range of adaptable subject distances.

[0345] Furthermore, in the imaging apparatus of the embodiment, the aperture changing mechanism has multiple light-passing sections (31-1, 31-2 and 31-3) with different thicknesses or sizes of the formed apertures, and is configured to switch the light-passing sections used for imaging.

[0346] By switching the light transmission section as described above, the thickness or size of the aperture used for imaging can be changed.

[0347] Since it does not require changing the thickness or size of the aperture itself, it is possible to stably control the switching of the aperture thickness or size used for imaging.

[0348] Furthermore, in the imaging apparatus implemented as an example, the aperture changing mechanism switches the light transmission section used for imaging using a turntable method.

[0349] Therefore, it is possible to switch the thickness or size of the aperture used for imaging while ensuring the sealing of the protective components. In particular, the turntable method can reduce the gaps generated in the protective components, which is suitable in terms of protective performance.

[0350] Furthermore, in the imaging apparatus implemented as an example, the aperture changing mechanism switches the light transmission section used for imaging by a sliding method.

[0351] Moreover, by using the sliding method, it is possible to switch the thickness or size of the hole used for imaging while ensuring the sealing of the protective components.

[0352] Note that the effects described in this specification are merely illustrative and not limited, and other effects may also be provided.

[0353] <7. This technology>

[0354] It should be noted that this technology can also be configured as follows.

[0355] (1) An imaging device, comprising: Imaging elements acquire captured images; and A protective component, at least covering the imaging surface of the imaging element to protect the imaging element, is configured to guide light from the outside to the imaging surface through a through hole in the thickness direction.

[0356] (2) According to the imaging device of (1) above, the protective component is a shielding component that shields the inside of the device from the outside, or a discharge and circulation component that has the function of external discharge or internal circulation to improve the internal environment of the device.

[0357] (3) The imaging device according to (1) or (2) above, wherein the reflection suppression component is formed in the area surrounding the hole in the protective component.

[0358] (4) An imaging device according to any one of (1) to (3) above, wherein a plurality of holes are formed.

[0359] (5) The imaging device according to (4) further includes: An image correction unit corrects the blurring in the captured image caused by the superposition of the positional offset of the images at each hole on the imaging plane.

[0360] (6) The imaging device according to (5) above, wherein the image correction unit uses an AI model to correct blur.

[0361] (7) An imaging apparatus according to any one of (1) to (6) above, wherein the sensor distance, which is the distance between the aperture and the imaging surface, is set such that the size of the subject light illumination range is substantially consistent with the size of the imaging range of the imaging element, the subject light illumination range being the range of subject light illuminating the imaging surface side through all the apertures.

[0362] (8) The imaging apparatus according to any one of (1) to (7) above further includes: Image correction unit, corrects blurring in captured images caused by lensless imaging; and The control unit controls the switching between multiple correction algorithms derived for each distance in the image correction unit for different subject distances.

[0363] (9) The imaging device according to (8) above, wherein the image correction unit uses an AI model to correct blur, and the control unit controls the AI ​​model used by the image correction unit to switch between multiple AI models derived for each distance in different subject distances.

[0364] (10) The imaging apparatus according to any one of (1) to (9) above further includes

[0365] Aperture changing mechanism, which is a mechanism for changing the thickness or size of the aperture used for imaging.

[0366] (11) Based on the imaging device described in (10) above, The aperture changing mechanism has multiple light-passing sections with different thicknesses or sizes of the formed aperture, and the aperture changing mechanism is configured to switch the light-passing sections used for imaging.

[0367] (12) Based on the imaging device described in (11) above, The aperture changing mechanism switches the light transmission section used for imaging via a turntable method.

[0368] (13) Based on the imaging device described in (11) above, The aperture changing mechanism switches the light transmission section used for imaging via a sliding method.

[0369] Reference Symbol List

[0370] 1. Imaging devices 1A and 1B

[0371] 2 Imaging elements

[0372] 3 Protective components

[0373] 3a Hole

[0374] 31 Light transmission section

[0375] 4 Circuit Units

[0376] 5. High thermal conductivity transparent components

[0377] 35 Substrate Department

[0378] 36 Hydrophobic layer

[0379] 10 Image Correction Units

[0380] 20 monitors

[0381] 1' Imaging device for generating learning data

[0382] 21 Learning Devices

[0383] 37 Reflection suppression components

[0384] 11, 11B Control Unit

[0385] 12 storage units

[0386] 13 Operation Units

[0387] M1 First AI Model

[0388] M2 Second AI Model

[0389] M3 Third AI Model

[0390] 40 Turntable Section

[0391] 31-1 First Light Transmission Section

[0392] 31-2 Second light transmission section

[0393] 31-3 Third Light Transmission Section

[0394] 15 Actuators

[0395] 41. Sliding part.

Claims

1. An imaging device, comprising: Imaging elements acquire captured images; as well as A protective component, at least covering the imaging surface of the imaging element to protect the imaging element, the protective component being configured to guide light from the outside to the imaging surface through a through hole in the thickness direction.

2. The imaging device according to claim 1, wherein, The protective component is a shielding component that isolates the interior of the device from the outside world, or a discharge and circulation component that has an external discharge or internal circulation function to improve the internal environment of the device.

3. The imaging device according to claim 1, wherein, A reflection suppression component is formed in the area surrounding the hole in the protective component.

4. The imaging device according to claim 1, wherein, Multiple holes are formed.

5. The imaging apparatus according to claim 4, further comprising: An image correction unit corrects the blurring in the captured image caused by the superposition of the positional offsets of the images of each hole on the imaging surface.

6. The imaging apparatus according to claim 5, wherein, The image correction unit uses an AI model to correct the blur.

7. The imaging apparatus according to claim 1, wherein, The sensor distance, which is the distance between the aperture and the imaging surface, is set such that the size of the subject light illumination range is substantially the same as the size of the imaging range of the imaging element, wherein the subject light illumination range is the range of subject light that is illuminated to the imaging surface side through all the apertures.

8. The imaging apparatus according to claim 1, further comprising: An image correction unit corrects the blurring in the captured image caused by lensless imaging. as well as The control unit controls the image correction unit to switch between multiple correction algorithms derived for each distance at different subject distances.

9. The imaging apparatus according to claim 8, wherein, The image correction unit uses an AI model to correct the blur, and The control unit controls the switching between multiple AI models derived by the AI ​​model used by the image correction unit for each distance at different subject distances.

10. The imaging apparatus according to claim 1, further comprising: Aperture changing mechanism, which is a mechanism for changing the thickness or size of the aperture used for imaging.

11. The imaging apparatus according to claim 10, wherein, The aperture changing mechanism has multiple light-passing sections with different thicknesses or sizes of the formed apertures, and the aperture changing mechanism is configured to switch the light-passing sections used for imaging.

12. The imaging apparatus according to claim 11, wherein, The aperture changing mechanism switches the light transmission section used for imaging via a turntable method.

13. The imaging apparatus according to claim 11, wherein, The aperture changing mechanism switches the light-passing section used for imaging via a sliding method.

Images