Optical devices and imaging devices

Abstract

To provide an optical device and an imaging apparatus that can generate optical information that is enlarged and on a wide range.SOLUTION: An optical device 21 comprises: an optical system 11 that forms first light incident thereon into an image in a predetermined area pa; an optical element 13 that guides, to the predetermined area pa, second light different from the first light in the angle formed by an optical axis of the optical system 11 and a principal ray incident on the optical system 11; and a prism 30 that is located in a space surrounded by the optical system 11, the optical element 13, and the predetermined area pa.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an optical device and an imaging device.

Background Art

[0002] An imaging optical system for forming an image of an observation target has various physical properties such as focal length and angle of view. When the focal length increases, an enlarged image of the observation target is formed, so that detailed optical information of a distant observation target, or in other words, enlarged optical information can be obtained. The wider the angle of view, the more optical information of observation targets located in a wide range can be obtained. However, there is a trade-off relationship between the focal length and the angle of view. When the focal length increases, the angle of view narrows, and when the focal length decreases, the angle of view widens.

[0003] Therefore, the focal length is adjusted so as to obtain desired optical information according to the situation. For example, the focal length is adjusted by displacing a zoom lens included in the imaging optical system. Also, the focal length is adjusted by switching a plurality of single-focus lenses (see Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Documents 1 and 2, enlarged optical information and wide-range optical information can be provided separately by switching the focal length. However, it was not possible to obtain wide-range and enlarged optical information simultaneously.

[0006] In view of the foregoing, the purpose of this disclosure is to provide an optical device and an imaging device that generate a wide range of expanded optical information. [Means for solving the problem]

[0007] An optical apparatus according to one embodiment of the present disclosure comprises an optical system, an optical element, and a prism. The optical system images an incident first light into a predetermined region. The optical element guides a second light into the predetermined region, the second light having a different angle between the optical axis of the optical system and the principal ray incident on the optical system than the first light. The prism is located in the space enclosed by the optical system, the optical element, and the predetermined region.

[0008] An imaging device according to one embodiment of the present disclosure comprises the optical device and an image sensor arranged such that the predetermined region and the imaging region overlap. [Effects of the Invention]

[0009] According to this disclosure, extensive and expanded optical information can be generated. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing the general configuration of the imaging device according to the first embodiment. [Figure 2] This figure shows a modified version of the optical element in Figure 1, viewed from a direction perpendicular to the optical axis. [Figure 3] This figure shows the imaging device viewed from a direction perpendicular to the optical axis, illustrating another modification of the optical element shown in Figure 1. [Figure 4] To illustrate yet another modification of the optical element in Figure 1, this is a view of the imaging device from a direction perpendicular to the optical axis. [Figure 5] To illustrate yet another modification of the optical element in Figure 1, this is a view of the imaging device from a direction perpendicular to the optical axis. [Figure 6] To illustrate yet another modification of the optical element in Figure 1, this is a view of the imaging device from a direction perpendicular to the optical axis. [Figure 7]It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 8] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 9] It is a diagram for explaining the physical properties of the imaging element and the optical system of FIG. 1. [Figure 10] It is a conceptual diagram for explaining the image reaching the light-receiving region of FIG. 1. [Figure 11] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 12] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 13] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 14] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 15] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 16] It is a view of the imaging device seen from the normal direction of the light-receiving region in order to show yet another modification example of the optical element of FIG. 1. [Figure 17] It is a conceptual diagram for explaining the situation where a superimposed image is formed in the light-receiving region of FIG. 1. [Figure 18] It is a conceptual diagram for explaining the process of generating a restored image from the superimposed image by the controller of FIG. 1. [Figure 19] It is a flowchart for explaining the distance measurement process executed by the controller of FIG. 1. [Figure 20] It is a diagram showing a schematic configuration example of the imaging device according to the second embodiment. [Figure 21] It is a diagram showing an example of a superimposed image of images formed by each of two imaging optical systems. [Figure 22]This figure shows an example of directly imaging an object point outside the field of view in the configuration shown in Figure 20. [Figure 23] This figure shows a schematic configuration example of an imaging device according to the third embodiment. [Figure 24] This is a schematic diagram showing the general configuration of the imaging device according to the fourth embodiment. [Figure 25] This is a conceptual diagram illustrating the image components that reach the light-receiving region in Figure 24. [Figure 26] This is a conceptual diagram illustrating the image components that reach the light-receiving region in a modified example of Figure 20. [Figure 27] This is a conceptual diagram illustrating the image components that reach the light-receiving region in another modified example shown in Figure 20. [Figure 28] This is a schematic diagram showing the general configuration of the imaging device according to the fifth embodiment. [Figure 29] Figure 28 is a diagram illustrating the pixel structure in the image sensor. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described below with reference to the drawings. In the components shown in the following drawings, the same components are denoted by the same reference numerals.

[0012] As shown in Figure 1, the imaging device 10 according to the first embodiment of this disclosure comprises an optical device 21, an image sensor 12, and a prism 30. The imaging device 10 may further comprise a controller 14. The optical device 21 comprises an imaging optical system 11 and an optical element 13. The imaging optical system 11 is also simply referred to as the optical system.

[0013] The imaging optical system 11 forms an image of light or a beam of light incident from a subject. The light or beam of light incident from a subject is also called subject light or subject beam of light. The imaging optical system 11 forms an image of the first incident light in a predetermined region pa. The first light may be light emitted from an object point located within the field of view of the imaging optical system 11 alone. The predetermined region pa may be, for example, a virtual plane or curved surface in three-dimensional space whose center intersects the optical axis ox of the imaging optical system 11. Hereafter, the field of view of the imaging optical system 11 alone, in other words, the field of view of the imaging optical system 11 in a configuration that does not include the optical elements 13, will also be called the direct field of view. The imaging optical system 11 is composed of optical elements that, individually, or in other words without the optical elements 13, form images of light or beams of light emitted from object points at different positions into different image points. The optical elements that make up the imaging optical system 11 are, for example, lenses, mirrors, diaphragms, etc.

[0014] The imaging optical system 11 does not have to be image-side telecentric. In other words, the angle of the principal ray of any light beam passing through the imaging optical system 11 with respect to the optical axis may be greater than 0°. Alternatively, the imaging optical system 11 may be image-side telecentric.

[0015] The optical element 13 guides the second light incident on the imaging optical system 11 to a predetermined region pa. The angle between the optical axis ox of the imaging optical system 11 and the principal ray incident on the imaging optical system 11 of the second light is different from that of the first light. The second light may be light emitted from an object point located outside the field of view of the imaging optical system 11, in other words, directly outside the field of view. Therefore, the angle between the principal ray of the second light and the optical axis ox may be larger than the angle between the principal ray of the first light and the optical axis ox. The principal ray may be a ray passing through the center of the aperture diaphragm of the imaging optical system 15, a ray passing through the center of the entrance pupil of the imaging optical system 15, or a ray at the center of a light beam emitted from any one object point and incident on the imaging optical system 15. Furthermore, the optical element 13 may image the second light that has passed through the imaging optical system 11 into a predetermined region pa.

[0016] The optical element 13 may be a mirror that reflects the second light and guides it to a predetermined region pa. The reflective surface of the mirror may be parallel to the optical axis ox of the imaging optical system 11. Alternatively, the reflective surface of the mirror may not be parallel to the optical axis ox.

[0017] The prism 30 is located between the imaging optical system 11 and the image sensor 12, and between the two optical elements 13. In other words, the prism 30 is located in the space enclosed by the imaging optical system 11, the optical elements 13, and a predetermined region pa. The prism 30 is made of a transparent material such as glass or resin. The prism 30 has a first surface 31 facing the imaging optical system 11. The prism 30 has a second surface 32 facing the optical elements 13. The prism 30 has a third surface 33 facing the image sensor 12.

[0018] Light or a beam of light from a subject that has passed through the imaging optical system 11 enters the first surface 31 of the prism 30, passes through the prism 30, and enters the image sensor 12. The image sensor 12 has a light-receiving area ra. The image sensor 12 images the light that enters the light-receiving area ra. The light-receiving area ra is also called the imaging area. The imaging optical system 11 is configured to image the subject light or subject beam onto the light-receiving area ra of the image sensor 12, taking into account the refraction of light at the first surface 31 of the prism 30. The optical element in the imaging optical system 11 that is closest to the image sensor 12 may be configured integrally with the first surface 31 of the prism 30. The prism 30 may be included in the imaging optical system 11 and function as the optical element in the imaging optical system 11 that is closest to the image sensor 12.

[0019] The prism 30 contacts the reflective surface of the mirror constituting the optical element 13 at its second surface 32. The second surface 32 of the prism 30 may be integrally formed with the optical element 13. In other words, the second surface 32 of the prism 30 may function as the reflective surface of the mirror constituting the optical element 13. When the second surface 32 of the prism 30 functions as the reflective surface of a mirror, the surface of the transparent member may be coated with a material having high reflectivity, such as metal or resin.

[0020] The prism 30 may have a surface on its third surface 33 that contacts the surface of the cover of the image sensor 12. The prism 30 may be integrally formed with the cover of the image sensor 12.

[0021] The reflective surface of the mirror may be tilted with respect to the optical axis ox so that it faces outward towards the imaging optical system 11, as shown in Figure 2. In this outward tilted position, the field of view of the entire optical device 21 can be widened compared to a configuration in which the reflective surface of the mirror is parallel to the optical axis ox. Even in this case, the prism 30 may be positioned between the reflective surfaces of the mirror of the optical element 13.

[0022] In an outward-tilting configuration, the optical device 21 may have a first lens 22 for adjusting the optical path length positioned between the imaging optical system 11 and the optical element 13, which is a mirror. By positioning the first lens 22, the deviation of the focus position from a predetermined region pa due to a longer optical path length compared to a configuration in which the reflective surface of the mirror is parallel to the optical axis ox can be reduced. In a configuration in which the mirror is a mirror having a surface parallel to the direction perpendicular to the optical axis ox, such as a planar mirror, the first lens 22 may be a cylindrical lens. The first lens 22 may be positioned outward from the optical axis ox than the line connecting the principal ray of the first light passing through the outer edge of the exit pupil of the imaging optical system 11 and the predetermined region pa.

[0023] Furthermore, in the outward-tilting configuration, the optical device 21 may be provided with a prism 23, as shown in Figure 3. The second light may be reflected by the optical element 13, which is a mirror, and then reflected again by the prism 23, thereby being guided to a predetermined region pa. By providing the prism 23, the tilt angle between the mirror and the optical axis ox in the outward-tilting configuration can be widened. Even in this case, the prism 30 may be installed on top of the image sensor 12. The prism 23 may be positioned so as to be in contact with at least a part of the prism 30. The prism 23 may be configured integrally with the prism 30.

[0024] In the outward-tilting configuration, the optical element 13, which is a mirror, may be, for example, a planar mirror, a curved mirror, a DMD (Digital Mirror Device), or a Fresnel mirror.

[0025] The reflective surface of the mirror may be tilted with respect to the optical axis ox so as shown in Figure 4, that it faces inward towards the imaging plane of the imaging optical system 11. In this inward tilted position, the entire optical device 21 can be made smaller compared to a configuration where the reflective surface of the mirror is parallel to the optical axis ox. Even in this case, the prism 30 may be positioned between the reflective surfaces of the mirrors of the optical elements 13.

[0026] In the inwardly tilted configuration, the optical element 13, which is a mirror, may be, for example, a planar mirror, a curved mirror as shown in Figure 5, a DMD, or a Fresnel mirror as shown in Figure 6. The prism 30 may be positioned between the reflective surfaces of the mirror of the optical element 13 in Figure 5. The prism 30 may also be positioned between the DMD or Fresnel mirror in Figure 6. If the mirror in Figure 6 is a DMD, the prism 30 may be positioned so as not to obstruct the operation of the DMD.

[0027] The reflective surface of the mirror may be parallel to any side of the rectangular light-receiving region ra of the image sensor 12, as described later. Alternatively, the reflective surface of the mirror may intersect with any side of the light-receiving region ra, as shown in Figure 7. In this case as well, the prism 30 may be positioned along the reflective surface of the mirror or in contact with the reflective surface of the mirror. In the configuration where the reflective surface of the mirror intersects with any side of the light-receiving region ra, the separation accuracy of the image separation model described later can be improved. In the configuration where the reflective surface of the mirror intersects with any side of the light-receiving region ra, it is preferable to position the optical element 13 such that the overlapping region between the region sandwiched between two straight lines extending perpendicularly from both ends of the optical element 13 (which is the mirror) and the light-receiving region ra is maximized when viewed from the normal direction of the light-receiving region ra.

[0028] The mirror may be located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction of the imaging optical system 11. More specifically, the mirror may be positioned relative to the imaging optical system 11 such that its reflective surface is located outside the exit pupil. Alternatively, the mirror may be located inside the exit pupil when viewed from the optical axis ox direction. In particular, in configurations where the light-receiving area ra is smaller than the pupil diameter, the mirror may be located inside the exit pupil.

[0029] The mirror may include multiple planar mirrors. Two planar mirrors belonging to at least one pair of the multiple planar mirrors may be positioned so that their reflective surfaces face each other and are parallel. Alternatively, the multiple planar mirrors may consist of two planar mirrors, positioned so that their reflective surfaces are perpendicular to each other, as shown in Figure 8. Furthermore, two planar mirrors whose reflective surfaces are perpendicular to each other may be parallel to two mutually perpendicular sides of the rectangular light-receiving region ra. The planar mirror and the outer edge of the light-receiving region ra of the image sensor 12 may be in close contact in the direction normal to the planar mirror. Alternatively, there may be a gap between the planar mirror and the outer edge of the light-receiving region ra, and they may not be in close contact in the direction normal to the planar mirror. Even in this case, the prism 30 may be positioned along the reflective surface of the mirror or in contact with the reflective surface of the mirror.

[0030] As shown in Figure 9, the distance H between each of the two parallel planar mirrors and the optical axis ox may be equal. The two parallel planar mirrors, the imaging optical system 11, and the image sensor 12 may be designed and arranged such that CRA ≤ tan⁻¹(H / B). CRA is the angle of the principal ray of the light beam emitted from the object point pp at an angle twice the angle of direct view with respect to the optical axis ox by the imaging optical system 11. B is the back focus of the imaging optical system 11. Even in this case, the prism 30 may be located between the reflective surfaces of the mirrors of the optical element 13.

[0031] As will be described later, due to the arrangement of the image sensor 12 within the imaging device 10 and the configuration described above, as shown in Figure 10, the first image component im1 corresponding to the first light reaches the light-receiving region ra of the image sensor 12 without passing through the optical element 13. More specifically, the first image component im1 corresponding to the first light corresponds to the image of the subject located directly within the field of view. In addition, the second image component im2 corresponding to the second light reaches the light-receiving region ra inverted via the optical element 13. More specifically, the second image component im2 corresponding to the second light corresponds to the image of the subject located directly outside the field of view.

[0032] In the above description, the optical element 13 is a mirror having a surface parallel to the direction perpendicular to the optical axis ox, but it may be a mirror having a curved surface when viewed from the optical axis ox. For example, as shown in Figure 11, the optical element 13 may be a pair of curved mirrors provided on opposite sides of a pair of rectangular light-receiving regions ra when viewed from the normal direction of the light-receiving region ra. The curved mirrors may be parallel to the normal direction of the light-receiving region ra. Alternatively, as shown in Figure 12, the optical element 13 may be a mirror having a circular curved surface that encloses the rectangular light-receiving region ra when viewed from the normal direction of the light-receiving region ra. Alternatively, as shown in Figure 13, the optical element 13 may be a mirror having an elliptical curved surface that encloses the rectangular light-receiving region ra when viewed from the normal direction of the light-receiving region ra. A mirror having an elliptical curved surface is preferred in configurations where the light-receiving region ra is a rectangle other than a square. Alternatively, the optical element 13 may be a mirror having a circular curved surface enclosed within a rectangular light-receiving region ra, as shown in Figure 14, when viewed from the normal direction of the light-receiving region ra. Alternatively, the optical element 13 may be a mirror having an elliptical curved surface enclosed within a rectangular light-receiving region ra, as shown in Figure 15, when viewed from the normal direction of the light-receiving region ra. In a configuration where the optical element 13 is a mirror having a curved surface enclosed within a rectangular light-receiving region ra, the gap between the light-receiving region ra and the mirror can be eliminated when viewed from the normal direction of the light-receiving region ra. In such a configuration, the elimination of the gap can improve the continuity of optical information in the superimposed image, as described later, compared to a configuration with a gap. Even when the mirror has a curved surface, the prism 30 may be arranged along the reflective surface of the mirror or in contact with the reflective surface of the mirror. The outer shape of the prism 30 may be shaped to match the shape of the reflective surface of the mirror.

[0033] The image sensor 12 captures an image that is formed within the light-receiving area ra. The image sensor 12 may be positioned in the imaging device 10 such that the light-receiving area ra overlaps with a predetermined area pa of the optical device 21. Therefore, the light-receiving area ra of the image sensor 12 may correspond to the direct field of view. The direct field of view may be the field of view corresponding to the range of object points that are imaged within the light-receiving area ra without passing through the optical element 13. At least a portion of the light beam, which is the first light, that is incident on the imaging optical system 11 from within the direct field of view of the imaging optical system 11 may be imaged in the light-receiving area ra. In addition, at least a portion of the light beam, which is the second light, that is incident on the imaging optical system 11 from outside the direct field of view of the imaging optical system 11 and passes through the optical element 13 may be imaged in the light-receiving area ra.

[0034] The image sensor 12 may be capable of capturing images using invisible light such as visible light, infrared light, and ultraviolet light. The image sensor 12 may be, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. The image sensor 12 may be a color image sensor. In other words, the multiple pixels arranged in the light-receiving area ra of the image sensor 12 may be covered by, for example, an RGB (Red, Green, and Blue) color filter so as to be evenly distributed within the light-receiving area ra. The light-receiving area ra of the image sensor 12 may also be covered by a cover. The cover may be made of a transparent material such as glass or resin. The image sensor 12 generates an image signal corresponding to the image received by imaging. The image sensor 12 may generate the image signal at a predetermined frame rate such as 30 fps (frames per second).

[0035] In the image sensor 12, the outer edge of the light-receiving region ra on the side where the optical element 13 is provided may be located outside the outer edge of the exit pupil of the imaging optical system 11. Outside the outer edge of the exit pupil means outside with respect to the optical axis ox of the imaging optical system 11. As mentioned above, the light-receiving region ra may be rectangular.

[0036] The imaging device 10 may be provided with a plurality of image sensors 12. In a configuration where a plurality of image sensors 12 are provided, an optical element 13 may be provided between two adjacent image sensors 12, as shown in Figure 16. By providing an optical element 13 between two adjacent image sensors 12, subject light or subject light beam that would be imaged in the gap between the light-receiving areas ra of two adjacent image sensors 12 in a configuration where the optical element 13 is not provided can be imaged by at least one of the image sensors 12. The prism 30 may be placed on the image sensor 12 on which the optical element 13 is provided, along the reflective surface of the mirror, or in contact with the reflective surface of the mirror. The prism 30 may also be placed on the image sensor 12 on which the optical element 13 is not provided. The prism 30 may be placed across the image sensor 12 on which the optical element 13 is provided and the image sensor 12 on which the optical element 13 is not provided.

[0037] Furthermore, with the above-described configuration, as shown in Figure 17, the first image component im1 and the second image component im2, which is inverted in the configuration where the optical element 13 is a mirror, are superimposed in the light-receiving region ra. Therefore, the image sensor 12 captures an image olim which is the superimposed image of the first image component im1 and the second image component im2, which is inverted in the configuration where the optical element 13 is a mirror.

[0038] The controller 14 comprises at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The controller 14 may perform image processing on the image signal acquired from the image sensor 12.

[0039] As shown in Figure 18, the controller 14 may perform image processing to separate the superimposed image olim, which corresponds to the image signal, into a first image component im1 and a second image component im2. The controller 14 separates the superimposed image olim by applying an image processing method such as independent component analysis, wavelet method, or image separation model. The image separation model is, for example, a model constructed by creating a superimposed image by overlapping multiple images in advance and training the model with multiple images as the correct answers for this superimposed image. The image separation model may be a model that applies Pix-to-Pix, which generates paired images that reflect the relationship between a generator that generates images and a discriminator that determines whether the generated image is a false image, as in an Encoder-Decoder model. The controller 14 may generate a reconstructed image rcim by combining the separated first image component im1 and second image component im2.

[0040] The controller 14 may use the reconstructed image rcim to measure the distance of the subject captured around the imaging device 10. For example, the controller 14 may use the reconstructed image rcim to measure the distance based on DFD (Depth From Defocus). The controller 14 may use the reconstructed image rcim to measure the distance based on motion disparity (SLAM: Simultaneous Localization and Mapping, Motion Stereo), a separation model based on Deep Learning, or the ground-level distance measurement method. The ground-level distance measurement method is a method that calculates 3D coordinates based on image coordinates, assuming that the lower end of the subject image is located on the ground.

[0041] The controller 14 may generate a distance image based on the distance corresponding to each address in the reconstructed image rcim. The distance image is an image in which the pixel value of each pixel corresponds to the distance. The controller 14 may attach the distance image to an external device.

[0042] Next, the distance measurement process performed by the controller 14 in this embodiment will be explained using the flowchart in Figure 19. The distance measurement process starts each time an image signal is acquired from the image sensor 12.

[0043] In step S100, the controller 14 separates the second image component im2 from the superimposed image olim, which corresponds to the acquired image signal. After separation, the process proceeds to step S101.

[0044] In step S101, the controller 14 generates the first image component im1 by subtracting the second image component im2, which was separated in step S100, from the superimposed image olim. After generating the first image component im1, the process proceeds to step S102.

[0045] In step S102, the controller 14 generates a reconstructed image rcim by combining the second image component im2, which was separated in step S100, with the first image component im1, which was generated in step S101. After the generation of the reconstructed image rcim, the process proceeds to step S103.

[0046] In step S103, the controller 14 uses the reconstructed image rcim generated in step S102 to measure the distance to each subject in the reconstructed image rcim. After measuring the distance, the process proceeds to step S104.

[0047] In step S104, the controller 14 generates a distance image based on the distance calculated in step S103 and the position of the reconstructed image rcim corresponding to that distance. The controller 14 also assigns the distance image to an external device. After the distance image is generated, the distance measurement process is terminated.

[0048] The optical device 21 of the first embodiment, having the configuration described above, includes an imaging optical system 11 that images an incident first light into a predetermined region pa, and an optical element 13 that guides a second light, whose angle between the optical axis ox of the imaging optical system 11 and the principal ray incident on the imaging optical system 11 is different from that of the first light, into the predetermined region pa. With this configuration, the optical device 21 can guide an image containing optical information in a wider angle range than the angle of view corresponding to the focal length into a predetermined region ra, even while employing an imaging optical system 11 with a relatively long focal length. Therefore, the optical device 21 can generate wide-range and magnified optical information.

[0049] Furthermore, in the imaging device 10 of the first embodiment, the optical element 13 is a mirror that reflects the second light and guides it to a predetermined region pa, and the reflective surface of the mirror is parallel to the optical axis ox and to one of the sides of the rectangular light-receiving region ra of the imaging element 12. With this configuration, in the imaging device 10, the direction in which the image reaches the light-receiving region ra via the mirror is distorted is limited to one direction or less. Therefore, the imaging device 10 can reduce the image processing load required to remove distortion from the reflected light component due to the mirror included in the captured image, thereby improving the reproducibility of the reflected light component.

[0050] Furthermore, the imaging device 10 of the first embodiment includes a prism 30. The second surface 32 of the prism 30 is in contact with the reflective surface of the optical element 13, or is integrally formed with the reflective surface of the optical element 13. With this configuration, the reflective surface of the optical element 13 is less likely to move relative to the imaging optical system 11. In other words, the position of the optical element 13 is stable. As a result, the effect of displacement of the optical element 13 due to vibration or shock is reduced.

[0051] Furthermore, in the imaging device 10 of the first embodiment, the mirror includes a plurality of planar mirrors, and at least one pair of the plurality of planar mirrors is positioned so that their reflective surfaces are parallel to each other. With this configuration, the imaging device 10 can acquire optical information that is expanded on both sides from the direct field of view, centered on the optical axis ox.

[0052] Furthermore, in the imaging device 10 of the first embodiment, the distance H between the optical axis ox and the reflective surfaces of the two planar mirrors parallel to each other is equal, and the angle CRA of the principal ray of the light beam from the object point pp at an angle twice the angle of direct view is CRA ≤ tan -1 The (H / B) ratio is satisfied. With this configuration, the imaging device 10 prevents the superposition of three layers of image components due to the superposition of the reflected light component from one plane mirror and the reflected light component from the other plane mirror, thereby improving the accuracy of image component separation in subsequent image processing.

[0053] Furthermore, in the imaging device 10 of the first embodiment, the plane mirror and the outer edge of the light-receiving area ra of the image sensor 12 are in close contact in the direction normal to the plane mirror. If a gap occurs between the plane mirror and the light-receiving area ra of the image sensor 12 in the direction normal to the plane mirror, the optical information of the subject being imaged in that gap will be lost. The imaging device 10 having the above configuration prevents such loss of optical information.

[0054] Furthermore, in the imaging device 10 of the first embodiment, the mirror is located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. With this configuration, the imaging device 10 can cause light or light beams passing near the end of the exit pupil to be incident on the mirror. Therefore, the imaging device 10 can reduce the reduction in light intensity due to vignetting of a portion of the light or light beam passing near the exit pupil.

[0055] Furthermore, in the imaging device 10 of the first embodiment, the angle of the principal ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. With this configuration, since the imaging optical system 11 is not image-side telecentric in the imaging device 10, light or light beams from object points at angles wider than the direct field of view can be incident on the optical element 13. Therefore, the imaging device 10 can reliably generate optical information in a range widened from the direct field of view.

[0056] Furthermore, the imaging device 10 of the first embodiment includes a controller 14 that separates the image corresponding to the image signal into a first image component im1 corresponding to the first light and a second image component im2 corresponding to the second light. With this configuration, the imaging device 10 can generate an image in which the superposition of a superimposed image olim, in which multiple image components are superimposed, is eliminated.

[0057] Next, an imaging apparatus according to a second embodiment of this disclosure will be described. The second embodiment differs from the first embodiment in that two imaging optical systems 11 are arranged for one image sensor 12. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that parts having the same configuration as in the first embodiment will be denoted by the same reference numerals.

[0058] The imaging device according to the second embodiment generates a parallax image of an object by forming a parallax image of the object using each of the two imaging optical systems 11 and capturing the parallax image with the image sensor 12. The parallax image of the object is used to calculate the distance to each point of the object.

[0059] As shown in Figure 20, the imaging device 102 according to the second embodiment includes an imaging optical system 11 comprising a first imaging optical system 11A and a second imaging optical system 11B. The first imaging optical system 11A is also referred to as the first optical system. The second imaging optical system 11B is also referred to as the second optical system. The first optical system and the second optical system may be located in parallel. The imaging device 102 further comprises a prism 30 and an optical element 13. The prism 30 may be configured to be the same as or similar to the prism 30 in the imaging device 10 according to the first embodiment. The optical element 13 may be configured to be the same as or similar to the optical element 13 in the imaging device 10 according to the first embodiment.

[0060] The first imaging optical system 11A and the second imaging optical system 11B image the light or light beam incident from an object point onto the light-receiving region ra of the image sensor 12. Here, we assume that the imaging device 102 is imaging an object having object points pp1 and pp2 at both ends. In this case, the first imaging optical system 11A images object point pp1 onto imaging point ip1 on the light-receiving region ra, and images the other part of the object to the left of imaging point ip1. The second imaging optical system 11B images object point pp2 onto imaging point ip2 on the light-receiving region ra, and images the other part of the object to the right of imaging point ip2. As a result, the image of the object formed by the first imaging optical system 11A and the image of the object formed by the second imaging optical system 11B superimpose each other in the central region of the light-receiving region ra.

[0061] As shown in Figure 21, the image formed by the first imaging optical system 11A on the object is represented as the first image imA. The image formed by the second imaging optical system 11B on the object is represented as the second image imB. The first image imA includes a first non-superimposed image im_A1 that does not superimpose on the second image imB on the light-receiving region ra, and a first superimposed image im_A2 that superimposes on the second image imB on the light-receiving region ra. The second image imB includes a second non-superimposed image im_B1 that does not superimpose on the first image imA on the light-receiving region ra, and a second superimposed image im_B2 that superimposes on the first image imA on the light-receiving region ra.

[0062] The image sensor 12 captures a first non-superimposed image im_A1, a second non-superimposed image im_B1, and an image obtained by superimposing the first superimposed image im_A2 and the second superimposed image im_B2, thereby generating a superimposed image olim.

[0063] The imaging device 102 may further include a controller 14. The controller 14 may separate the superimposed image olim from the superimposed image olim into the superimposed image im_A2 and the superimposed image im_B2, and generate the first image imA and the second image imB from the superimposed image olim.

[0064] As shown in Figure 22, when the object point pp2 is imaged by the first imaging optical system 11A, the image point ip2 may be located outside the light-receiving region ra. In other words, the object point pp2 may be located outside the direct field of view of the first imaging optical system 11A. Even in this case, the light or light beam imaged at the image point ip2 is reflected inward by the optical element 13 into the light-receiving region ra and superimposed on the light-receiving region ra to form an image. As a result, the superimposed image olim includes an image obtained by superimposing the image of the object point in the direct field of view and the image of the object point outside the direct field of view in the first image imA. Furthermore, when the object point pp1 is imaged by the second imaging optical system 11B, the image point ip1 may be located outside the light-receiving region ra. In other words, the object point pp1 may be located outside the direct field of view of the second imaging optical system 11B. Even in this case, the light or light beam imaged at the image point ip1 is reflected inward by the optical element 13 into the light-receiving region ra and superimposed on the light-receiving region ra to form an image. As a result, the superimposed image olim includes an image obtained by superimposing the images of object points within the direct field of view and the images of object points outside the direct field of view in the second image imB.

[0065] The controller 14 may separate the images of object points within the direct field of view from the images of object points outside the direct field of view for each of the superimposed images olim, namely the first image imA and the second image imB, and generate an image that extends to include object points outside the direct field of view.

[0066] As described above, the imaging device 102 according to the second embodiment can capture an image in which the first image imA and the second image imB are superimposed on the light-receiving area ra of the image sensor 12, and can separate the first image imA and the second image imB from the superimposed image olim. In this way, the imaging device 102 can capture a disparity image composed of two images with a single image sensor 12.

[0067] Furthermore, the imaging device 102 captures the parallax image with a wider field of view compared to simply dividing the light-receiving area ra into two areas and capturing a parallax image in each area, by capturing images such that the two images constituting the parallax image overlap each other. The imaging device 102 also includes an optical element 13 that images at least a portion of the light or light beam incident on the imaging optical system 11 from outside the direct field of view of the imaging optical system 11 corresponding to the light-receiving area ra of the image sensor 12 into the light-receiving area ra. With this configuration, the imaging device 10 can image an image containing optical information in a wider angle of view than the field of view corresponding to the focal length of the imaging optical system 11, even while employing an imaging optical system 11 with a relatively long focal length. Therefore, the imaging device 10 can generate wide-range and magnified optical information.

[0068] In distance measurement based on disparity images, the resolution and accuracy of the distance data can be improved as the baseline length increases. The baseline length corresponds to the distance between the devices that capture the two images that make up the disparity image. As a method for capturing disparity images to generate distance measurement data, the method described in the following comparative example can be considered.

[0069] As a first comparative example, a method using a stereo camera can be considered. A stereo camera is a method of triangulation that uses two cameras placed parallel to each other. In a stereo camera, the distance between the two cameras corresponds to the baseline length. Therefore, increasing the baseline length of the stereo camera can improve the resolution and accuracy of the distance data. In addition, by being able to set the focal length of each of the two cameras, the captured image can be widened. However, when increasing the baseline length in a stereo camera, the device becomes larger because the two cameras are placed further apart. Furthermore, calibration of the two cameras becomes necessary.

[0070] As a second comparative example, the pupil division method can be considered. The pupil division method is a method of constructing a stereo camera within the lens by dividing the pupil of the camera. The device related to the pupil division method can be made smaller compared to a stereo camera because it can be constructed with a single pupil. However, the baseline length is limited by the pupil diameter. As a result, it is difficult to increase the baseline length. Therefore, it is difficult to improve the resolution and accuracy of the distance data. Here, in the pupil division method, one method that can be considered to improve the resolution and accuracy of the distance data is to increase the focal length. However, increasing the focal length makes it difficult to widen the angle of the captured image. In other words, the limitation of the baseline length makes it difficult to widen the angle of the captured image or to improve the resolution and accuracy of the distance data.

[0071] As a third comparative example, a method can be considered in which the inputs of two pupils are superimposed and imaged with a single image sensor. A device using this method can have a longer baseline length and wider-angle images, similar to a stereo camera, and can be composed of fewer image sensors than a stereo camera. However, superimposing the inputs of two pupils requires a special optical design for the optical system. Furthermore, the number of components in the optical system increases. As a result, miniaturization becomes costly.

[0072] On the other hand, the imaging device 102 according to the second embodiment can capture two images with a single image sensor 12, thereby achieving a wider angle of view for captured images while being smaller than the stereo camera according to the first comparative example. Furthermore, by providing a first imaging optical system 11A and a second imaging optical system 11B for a single image sensor 12, the imaging device 102 according to the second embodiment can have a longer baseline length than the pupil division method according to the second comparative example, thereby improving the resolution and accuracy of distance data. In addition, the imaging device 102 according to the second embodiment can separate the images corresponding to each superimposed image from the superimposed image captured by superimposing and imaging the images on the light-receiving area ra of the image sensor 12, thereby being simpler and smaller than the method according to the third comparative example.

[0073] As described above, the imaging device 102 according to the second embodiment can be implemented with a simpler and more compact configuration than the comparative example described above, while achieving wider-angle imaging and improved resolution and accuracy of distance data.

[0074] As shown in Figure 23, the imaging device 103 according to the third embodiment includes a first prism 30A corresponding to the first imaging optical system 11A and a second prism 30B corresponding to the second imaging optical system 11B. The imaging device 103 also includes a first optical element 13A located on the side of the first prism 30A and a second optical element 13B located on the side of the second prism 30B. In the imaging device 103, the light-receiving region ra of the image sensor 12 is divided into a first light-receiving region raA where the first prism 30A is located and a second light-receiving region raB where the second prism 30B is located.

[0075] The first imaging optical system 11A images light or a beam of light incident from object point pp3 at the first imaging point ipA on the first light-receiving region raA. The first imaging optical system 11A may also image light or a beam of light incident from other object points outside the first light-receiving region raA. Light or a beam of light imaged outside the first light-receiving region raA is reflected by the first optical element 13A and imaged at a point on the first light-receiving region raA. In other words, the configuration combining the first imaging optical system 11A, the first prism 30A, and the first optical element 13A can image an object on the first light-receiving region raA, including images of object points outside the direct field of view of the first imaging optical system 11A.

[0076] The second imaging optical system 11B images light or a beam of light incident from object point pp3 at the second imaging point ipB on the second light-receiving region raB. The second imaging optical system 11B may also image light or a beam of light incident from other object points outside the second light-receiving region raB. Light or a beam of light imaged outside the second light-receiving region raB is reflected by the second optical element 13B and imaged at a point on the second light-receiving region raB. In other words, the configuration combining the second imaging optical system 11B, the second prism 30B, and the second optical element 13B can image an object on the second light-receiving region raB, including images of object points outside the direct field of view of the second imaging optical system 11B.

[0077] The imaging device 103 according to the third embodiment has two sets of configurations, each consisting of one imaging optical system 11, one prism 30, and an optical element 13 located on the side of the prism 30. By using an imaging device 103 that combines two sets of identical configurations in this way, one set of configurations can be applied universally, thereby reducing manufacturing man-hours or costs.

[0078] In the imaging device 103 according to the third embodiment, the image sensor 12 separates and images the image formed by the first imaging optical system 11A and the image formed by the second imaging optical system 11B. In this case, the superimposed image olim includes an image in which the images of object points directly within the field of view and the images of object points directly outside the field of view are superimposed only within the first image imA and the second image imB, respectively. As a result, the image separation model and other methods used to separate the superimposed image from the superimposed image olim in the imaging device 10 according to the first embodiment can be applied as is.

[0079] In the imaging device 103 according to the third embodiment, the first optical element 13A may be formed as a reflective film on the side surface of the first prism 30A, and the second optical element 13B may be formed as a reflective film on the side surface of the second prism 30B. In this case, the area occupied by the first optical element 13A and the second optical element 13B in the light-receiving region ra of the imaging device 12 becomes smaller. In other words, the area of ​​the region that does not belong to either the first light-receiving region raA or the second light-receiving region raB and cannot receive light becomes smaller. Conversely, the area of ​​the image that can be captured in the light-receiving region ra can be widened. As a result, the field of view of the imaging device 103 can be widened.

[0080] Next, an imaging apparatus according to the fourth embodiment of this disclosure will be described. In the fourth embodiment, the configuration of the optical elements and the separation process by the controller differ from those of the first embodiment. The fourth embodiment will be described below, focusing on the differences from the first embodiment. Note that parts having the same configuration as in the first embodiment will be denoted by the same reference numerals.

[0081] As shown in Figure 24, the imaging device 100 according to the fourth embodiment is configured similarly to the first embodiment, including an imaging optical system 11, an image sensor 12, a prism 30, and an optical element 130. The imaging device 100 may further include a controller 14. The structure and function of the imaging optical system 11 and the image sensor 12 in the fourth embodiment are the same as in the first embodiment. The structure of the controller 14 in the fourth embodiment is the same as in the first embodiment.

[0082] In the fourth embodiment, the optical element 130, similar to the first embodiment, images at least a portion of the light or light beam incident on the imaging optical system 11 from outside the direct field of view of the imaging optical system 11 into the light-receiving area ra of the image sensor 12. In the fourth embodiment, unlike the first embodiment, the optical element 130 performs optical processing on the incident light or light beam before emitting it.

[0083] Optical processing involves, for example, a change in the bandwidth of the incident light or light beam. Specifically, the optical element 130 attenuates light in the incident light or light beam that corresponds to a certain bandwidth of the multiple color filters covering the image sensor 12. Therefore, the optical element 130 images the light in the bandwidths excluding the affected bandwidth within the light-receiving region ra.

[0084] The optical element 130 may be a mirror that reflects light in a frequency band other than the frequency band of the color to be attenuated. For example, the optical element 130 attenuates R light and reflects GB light.

[0085] With the configuration described above, as shown in Figure 25, in the fourth embodiment, the first R image component im1r, the first G image component im1g, and the first B image component im1b, which directly correspond to object points within the field of view and correspond to the colors of all color filters, reach the light-receiving region ra without passing through the optical element 130. In addition, the second G image component im2g and the second B image component im2b, which directly correspond to object points outside the field of view and correspond to colors other than the attenuated color components, reach the light-receiving region ra via the optical element 130.

[0086] Alternatively, the optical processing may involve, for example, adding a brightness difference pattern to the incident light or light beam according to the incident position. Specifically, as shown in Figure 26, the optical element 130 has a surface in which the first region 190 and the second region 200 are distributed, for example, in a checkerboard pattern. The first region 190 attenuates the brightness of the incident light by a first attenuation rate and emits it. The first attenuation rate is greater than 0% and less than 100%. The second region 200 attenuates the brightness of the incident light by a second attenuation rate and emits it. The second attenuation rate is 0% or more and less than the first attenuation rate. Therefore, the optical element 130 imparts a brightness difference to the incident light or light beam according to the pattern of the first region 190 and the second region 200, and images it within the light-receiving region ra.

[0087] Alternatively, optical processing involves, for example, adding distortion to the image formed by the incident light or light beam within the light-receiving region ra. Specifically, as shown in Figure 27, the optical element 130 is a mirror having a cylindrical curved surface with an axis parallel to the optical axis, and it reflects the incident light or light beam to form a distorted image within the light-receiving region ra. More specifically, the optical element 130 is a distorted image that expands in the direction connecting the ends of the arc in the cross-section of the optical element, which is a mirror formed by a plane perpendicular to the optical axis.

[0088] In the fourth embodiment, the controller 14 may perform image processing similar to the first embodiment to separate the superimposed image olim, which corresponds to the image signal, into a first image component im1 and a second image component im2 by image processing. In the fourth embodiment, the controller 14 separates the superimposed image olim by an image processing method using an image separation model.

[0089] The image separation model in the fourth embodiment is described below. The image separation model is constructed by first generating a first image that has not undergone any optical processing, and a second image that is different from the first image, and then applying image processing equivalent to the optical processing performed by the optical element 130 to this second image and superimposing it onto the first image to create a superimposed image, and then training the superimposed image with the first image and the second image as the correct answers.

[0090] In a configuration where the optical processing involves a change in bandwidth, the first image is the RGB image component of an arbitrary image. Furthermore, in the same configuration, the second image is the GB image component of an image other than the aforementioned arbitrary image. In a configuration where the optical processing involves a change in bandwidth, the R image component of the arbitrary image can also be used for training in addition to the superimposed image.

[0091] In a configuration where the optical processing involves adding a contrast pattern, the first image is an arbitrary image. In this configuration, the second image is an image obtained by changing the brightness of an image other than the arbitrary image using the contrast pattern of the optical element 130.

[0092] In a configuration where the optical processing involves adding distortion, the first image is an arbitrary image. In this configuration, the second image is an image of a different image, reflected by a mirror with the same curved surface as the optical element 130.

[0093] In the fourth embodiment, the controller 14 may generate a reconstructed image rcim by combining the separated first image component im1 and the second image component im2, as in the first embodiment. In the fourth embodiment, the controller 14 may use the reconstructed image rcim to measure the distance of subjects captured around the imaging device 100, as in the first embodiment. In the fourth embodiment, the controller 14 may generate a distance image based on the distances corresponding to each address of the reconstructed image rcim, as in the first embodiment, and provide it to an external device.

[0094] The optical device 210 of the fourth embodiment, having the configuration described above, also includes an imaging optical system 11 that images the incident first light into a predetermined region pa, and an optical element 130 that guides a second light, whose angle between the optical axis ox of the imaging optical system 11 and the principal ray incident on the imaging optical system 11 is different from that of the first light, into the predetermined region pa. Therefore, the imaging device 100 can also generate wide-range and magnified optical information.

[0095] Furthermore, in the imaging device 100 of the fourth embodiment, the optical element 130 is a mirror that reflects the second light and forms an image within a predetermined region pa, and the reflective surface of the mirror is parallel to either the optical axis ox or one of the sides of the rectangular light-receiving region ra of the imaging element 12. Therefore, the imaging device 100 can also reduce the image processing load required to remove distortion from the reflected light component due to the mirror included in the captured image, thereby improving the reproducibility of the reflected light component.

[0096] Furthermore, in the imaging device 100 of the fourth embodiment, the mirror also includes a plurality of planar mirrors, and at least one pair of the plurality of planar mirrors are positioned so that their reflective surfaces are parallel to each other. Therefore, the imaging device 100 can also acquire optical information expanded on both sides from the direct field of view, centered on the optical axis ox.

[0097] Furthermore, in the imaging device 100 of the fourth embodiment, the distance H between the optical axis ox and the reflective surfaces of the two planar mirrors parallel to each other is equal, and the angle CRA of the principal ray of the light beam from the object point pp at an angle twice the direct field of view is CRA ≤ tan -1 The (H / B) ratio is satisfied. Therefore, even in the imaging device 100, the superposition of three image components due to the superposition of the reflected light component from one plane mirror and the reflected light component from the other plane mirror is prevented, and the accuracy of separating image components by subsequent image processing can be improved.

[0098] Furthermore, in the imaging device 100 of the fourth embodiment, the plane mirror and the outer edge of the light-receiving area ra of the image sensor 12 are in close contact in the direction normal to the plane mirror. Therefore, the imaging device 100 also prevents the loss of optical information.

[0099] Furthermore, in the imaging device 100 of the fourth embodiment, the mirror is located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. Therefore, the imaging device 100 can also reduce the decrease in light intensity due to vignetting of a portion of the light or light beam passing near the exit pupil.

[0100] Furthermore, in the imaging device 100 of the fourth embodiment, the angle of the principal ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. Therefore, the imaging device 100 can also reliably generate optical information in a range widened from the direct field of view.

[0101] Furthermore, the imaging device 100 of the fourth embodiment also includes a controller 14 that separates the image corresponding to the image signal into a first image component im1 corresponding to the first light and a second image component im2 corresponding to the second light. Therefore, the imaging device 100 can also generate an image in which the superposition of a superimposed image olim, in which multiple image components are superimposed, is resolved.

[0102] Furthermore, in the imaging device 100 of the fourth embodiment, the optical element 130 performs optical processing on the light or light beam incident on the optical element 130 before emitting it. With this configuration, the imaging device 100 imparts optical features corresponding to the optical processing to the separated image components. Therefore, the imaging device 100 can construct an image separation model that has been learned to improve separation accuracy. Consequently, the imaging device 100 can improve the reconstruction accuracy of the reconstructed image.

[0103] Next, an imaging apparatus according to the fifth embodiment of this disclosure will be described. In the fifth embodiment, the configuration of the image sensor and the separation process by the controller differ from those of the first embodiment. The fifth embodiment will be described below, focusing on the differences from the first embodiment. Note that parts having the same configuration as in the first embodiment will be denoted by the same reference numerals.

[0104] As shown in Figure 28, the imaging device 101 according to the fifth embodiment is configured similarly to the first embodiment, including an imaging optical system 11, an image sensor 121, a prism 30, and an optical element 13. The imaging device 101 may further include a controller 14. The structure and function of the imaging optical system 11 and the optical element 13 in the fifth embodiment are the same as in the first embodiment. The structure of the controller 14 in the fifth embodiment is the same as in the first embodiment.

[0105] In the fifth embodiment, the image sensor 121 captures an image formed within the light-receiving region ra via the imaging optical system 11, similar to the first embodiment. The image sensor 121 may be capable of capturing images using invisible light such as visible light, infrared light, and ultraviolet light, similar to the first embodiment. The image sensor 121 may be a color image sensor. The image sensor 121 generates an image signal corresponding to the image received by imaging, similar to the first embodiment. The image sensor 121 may generate the image signal at a predetermined frame rate such as 30 fps, similar to the first embodiment. In the image sensor 121, as in the first embodiment, the outer edge of the light-receiving region ra on the side where the optical element 13 is provided may be located outside the outer edge of the exit pupil of the imaging optical system 11. The light-receiving region ra may be rectangular, as in the first embodiment.

[0106] In the fifth embodiment, the image sensor 121 may be a dual-pixel type image sensor, unlike in the first embodiment. As shown in Figure 29, the image sensor 121, which is a dual-pixel type image sensor, has a first PD (Photo Diode) 171 and a second PD 181 provided in each pixel 161 covered by each microlens 151, and is an image sensor having a structure in which light or a light beam can be incident on only one of the PDs depending on the direction of incidence of the light or light beam. For example, in each pixel 161, only light or a light beam from a direction inclined toward the optical axis ox can be incident on the first PD 171, and only light or a light beam from a direction inclined toward the optical element 13 can be incident on the second PD 181.

[0107] With the configuration described above, the first image component im1, corresponding to the first light, reaches the first PD171 in the light-receiving region ra without passing through the optical element 13. In addition, the second image component im2, corresponding to the second light, reaches the second PD181 in the light-receiving region ra after being inverted via the optical element 13.

[0108] In the fifth embodiment, the controller 14 may perform image processing similar to the first embodiment to separate the superimposed image olim, which corresponds to the image signal, into a first image component im1 and a second image component im2 by image processing. In the fifth embodiment, the controller 14 may generate the first image component im1 based only on the signal generated by the first PD 171. Based only on the signal generated by the first PD 171 means that the signal generated by the second PD 181 is not used, and may include using a signal unrelated to the signal output by the second PD 181, such as a synchronization signal. Also in the fifth embodiment, the controller 14 may generate an inverted second image component im2 based only on the signal generated by the second PD 181. Based only on the signal generated by the second PD 181 has a similar meaning to based only on the signal generated by the first PD 171.

[0109] In the fifth embodiment, the controller 14 may generate a reconstructed image rcim by combining the separated first image component im1 and the second image component im2, as in the first embodiment. In the fifth embodiment, the controller 14 may use the reconstructed image rcim to measure the distance of subjects captured around the imaging device 101, as in the first embodiment. In the fifth embodiment, the controller 14 may generate a distance image based on the distances corresponding to each address of the reconstructed image rcim, as in the first embodiment, and provide it to an external device.

[0110] In the imaging device 101 of the fifth embodiment with the configuration described above, the optical element 13 is a mirror that reflects the second light and guides it into a predetermined region pa, and the reflective surface of the mirror is parallel to either the optical axis ox or one of the sides of the rectangular light-receiving region ra of the imaging element 121. Therefore, the imaging device 101 can also reduce the image processing load required to remove distortion of the reflected light component due to the mirror included in the captured image, thereby improving the reproducibility of the reflected light component.

[0111] Furthermore, in the imaging device 101 of the fifth embodiment, the mirror also includes a plurality of planar mirrors, and at least one pair of the plurality of planar mirrors are positioned so that their reflective surfaces are parallel to each other. Therefore, the imaging device 101 can also acquire optical information that is expanded on both sides from the direct field of view, centered on the optical axis ox.

[0112] Furthermore, in the imaging device 101 of the fifth embodiment, the distance H between the optical axis ox and the reflective surfaces of the two planar mirrors parallel to each other is equal, and the angle CRA of the principal ray of the light beam from the object point pp at an angle twice the angle of direct view is CRA ≤ tan -1 The (H / B) ratio is satisfied. Therefore, even in the imaging device 101, the superposition of three image components due to the superposition of the reflected light component from one planar mirror and the reflected light component from the other planar mirror is prevented, and the accuracy of separating image components by subsequent image processing can be improved.

[0113] Furthermore, in the imaging device 101 of the fifth embodiment, the plane mirror and the outer edge of the light-receiving area ra of the image sensor 121 are in close contact in the direction normal to the plane mirror. Therefore, the imaging device 101 also prevents the loss of optical information.

[0114] Furthermore, in the imaging device 101 of the fifth embodiment, the mirror is located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. Therefore, the imaging device 101 can also reduce the decrease in light intensity due to vignetting of a portion of the light or light beam passing near the exit pupil.

[0115] Furthermore, in the imaging device 101 of the fifth embodiment, the angle of the principal ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. Therefore, the imaging device 101 can also reliably generate optical information in a range widened from the direct field of view.

[0116] Furthermore, the imaging device 101 of the fifth embodiment also includes a controller 14 that separates the image corresponding to the image signal into a first image component im1 corresponding to the first light and a second image component im2 corresponding to the second light. Therefore, the imaging device 101 can also generate an image in which the superposition of a superimposed image olim, in which multiple image components are superimposed, is eliminated.

[0117] The diagrams illustrating the embodiments described herein are schematic. Dimensions and proportions shown in the drawings do not necessarily correspond to actual dimensions.

[0118] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are within the scope of this disclosure. For example, the functions and other elements included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated.

[0119] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0120] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0121] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the components. Components distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, the first image component may swap the identifiers "First" and "Second" with the second image component. The swapping of identifiers occurs simultaneously. The components remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Components from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the components or to justify the existence of smaller numbered identifiers.

[0122] In one embodiment, (1) the optical device comprises an optical system that images an incident first light into a predetermined region, an optical element that guides a second light, whose angle between the optical axis of the optical system and the principal ray incident on the optical system is different from that of the first light, into the predetermined region, and a prism located in the space enclosed by the optical system, the optical element, and the predetermined region.

[0123] (2) In the optical apparatus described in (1) above, the optical element may include a reflective surface configured as at least one surface of the prism.

[0124] (3) In the optical apparatus described in (1) or (2) above, the angle between the principal ray of the second light and the optical axis may be greater than the angle between the principal ray of the first light and the optical axis.

[0125] (4) In any one of the optical devices described in (1) to (3) above, the optical element may include a mirror that reflects the second light and guides it to the predetermined region.

[0126] In one embodiment, (5) the optical device comprises an optical system that images an incident first light into a predetermined region, an optical element that guides a second light, whose angle between the optical axis of the optical system and the principal ray incident on the optical system is different from that of the first light, into the predetermined region, and a prism located in at least a part of the optical paths of the first light and the second light output from the optical system.

[0127] In one embodiment, (6) the imaging device comprises an optical device described in any one of (1) to (5) above, and an image sensor arranged so as to overlap the predetermined region and the imaging region.

[0128] (7) In the imaging device described in (6) above, the optical system may include a first optical system and a second optical system having different optical axes and having a portion of each subject in common. A portion of the image formed by the first optical system and a portion of the image formed by the second optical system may be superimposed in the light-receiving area of ​​the image sensor.

[0129] (8) In the imaging device described in (6) above, the optical system may include a first optical system and a second optical system having different optical axes and having a portion of each subject in common. The prism may include a first prism corresponding to the first optical system and a second prism corresponding to the second optical system.

[0130] (9) In any one of the imaging devices described in (6) to (8) above, the prism may be configured such that the reflective surface of the prism is parallel to the optical axis of the optical system.

[0131] (10) Any one of the imaging devices described in (6) to (9) above may further include a processor that separates the image captured by the image sensor into a first image component corresponding to the first light and a second image component corresponding to the second light.

[0132] While embodiments of imaging methods using imaging devices 10, 100, 101, 102, and 103 have been described above, embodiments of the present disclosure can also include not only methods or programs for implementing the apparatus, but also a storage medium on which the program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0133] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. In addition, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed. [Explanation of symbols]

[0134] 10, 100, 101, 102, 103 Imaging devices 11. Imaging optical system (11A: First imaging optical system, 11B: Second imaging optical system) 12, 121 Image sensor 13, 130 Optical elements (13A: First optical element, 13B: Second optical element) 14 Controllers 151 Microlenses 161 pixels 171 First PD (Photo-Diode) 181 Second PD 190 First Domain 200 Second Domain 21, 210, 211 Optical equipment 22 First lens 23 Prisms 30 prisms (30A: 1st prism, 30B: 2nd prism, 31: 1st surface, 32: 2nd surface, 33: 3rd surface) CRA (Critical Ray Analysis) Angle of the principal ray relative to the optical axis by the imaging optical system of the light beam emitted from an object point at an angle twice the direct field of view. im1 First image component im1b First B-image component im1g First G image component im1r First R image component im2: Second image component im2b Second B-image component im2g Second G image component imA First image (im_A1: First non-superimposed image, im_A2: First superimposed image) imB Second image (im_B1: second non-superimposed image, im_B2: second superimposed image) ip1, ip2 imaging points ipA, ipB: First imaging point, Second imaging point Olim superimposed image ox optical axis pa predetermined area rcim restored image pp Object point at an angle twice the direct field of view pp1, pp2, pp3 material points ra light receiving area (raA: 1st light receiving area, raB: 2nd light receiving area)

Claims

1. An optical system that images the incoming first light onto a predetermined light-receiving area of ​​an image sensor, An optical element that guides a second light incident on the optical system, wherein the angle between the optical axis of the optical system and the principal ray incident on the optical system is different from that of the first light, to the predetermined light-receiving region, and superimposes it on the image produced by the first light so that it is imaged by the image sensor, A prism located in the space enclosed by the optical system, the optical element, and the predetermined light-receiving region Prepare, An optical device that outputs an image signal to the image sensor obtained by superimposing the image obtained by the first light and the image obtained by the second light.

2. The optical apparatus according to claim 1, wherein the optical element includes a reflective surface configured as at least one surface of the prism.

3. The optical apparatus according to claim 1 or 2, wherein the angle between the principal ray of the second light and the optical axis is greater than the angle between the principal ray of the first light and the optical axis.

4. The optical device according to claim 1 or 2, wherein the optical element includes a mirror that reflects the second light and guides it to the predetermined light-receiving region.

5. An optical system that images the incoming first light onto a predetermined light-receiving area of ​​an image sensor, An optical element that guides a second light incident on the optical system, wherein the angle between the optical axis of the optical system and the principal ray incident on the optical system is different from that of the first light, to the predetermined light-receiving region, and superimposes it on the image produced by the first light so that it is imaged by the image sensor, A prism located in at least a portion of the optical paths of the first and second light beams output from the optical system, Equipped with, An optical device that outputs an image signal to the image sensor obtained by superimposing the image obtained by the first light and the image obtained by the second light.

6. The optical apparatus according to claim 1, 2, or 5, An image sensor arranged such that the predetermined light-receiving area and the imaging area overlap. An imaging device equipped with the following features.

7. An optical system that images the incoming first light into a predetermined region, An optical element that guides a second light, whose angle with respect to the optical axis of the optical system and the principal ray incident on the optical system is different from that of the first light, into the predetermined region, A prism located in the space enclosed by the optical system, the optical element, and the predetermined region, An image sensor arranged such that the predetermined region and the imaging region overlap. Equipped with, The optical system includes a first optical system and a second optical system, each having different optical axes and sharing a portion of the respective subject. An imaging device in which a portion of the image formed by the first optical system and a portion of the image formed by the second optical system are superimposed in the light-receiving area of ​​the image sensor.

8. An optical system that images the incoming first light into a predetermined region, An optical element that guides a second light, whose angle with respect to the optical axis of the optical system and the principal ray incident on the optical system is different from that of the first light, into the predetermined region, A prism located in the space enclosed by the optical system, the optical element, and the predetermined region, An image sensor arranged such that the predetermined region and the imaging region overlap. Equipped with, The optical system includes a first optical system and a second optical system, each having different optical axes and sharing a portion of the respective subject. The imaging device includes a first prism corresponding to the first optical system and a second prism corresponding to the second optical system.

9. The imaging apparatus according to claim 6, wherein the prism is configured such that the reflective surface of the prism is parallel to the optical axis of the optical system.

10. The imaging apparatus according to claim 6, further comprising a processor that separates the image captured by the image sensor into a first image component corresponding to the first light and a second image component corresponding to the second light.

Images