IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, AND IMAGE PROCESSING METHOD
The imaging device uses a reflective optical element to combine images from within and outside the direct angle of view, addressing the trade-off between focal length and angle of view, and enhances image reproducibility and accuracy.
Patent Information
- Application Number
- JP2023130261
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-10-27
- Filing Date
- 2023-08-09
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2042-08-23
AI Technical Summary
Existing imaging devices cannot simultaneously achieve wide-ranging and enlarged optical information due to the trade-off relationship between focal length and angle of view.
An imaging device that uses a reflective optical element to guide second light with a different angle to a predetermined area on the image sensor, allowing superposition and combination of images from within and outside the direct angle of view, with a controller to combine and separate these images effectively.
Enables the generation of extensive and expanded optical information, improving the reproducibility and accuracy of captured images by reducing distortion and enhancing the angle of view.
Smart Images

Figure 0007583874000001 
Figure 0007583874000002 
Figure 0007583874000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging device. [Background technology]
[0002] An imaging optical system that forms an image of an observation target has various physical properties such as focal length and angle of view. When the focal length is longer, an enlarged image of the observation target is formed, and detailed optical information of the distant observation target, in other words, enlarged optical information, can be obtained. As the angle of view becomes wider, optical information of an observation target located in a wider range can be obtained. However, there is a trade-off between focal length and angle of view, and as the focal length becomes longer, the angle of view becomes narrower, and as the focal length becomes shorter, the angle of view becomes wider.
[0003] Therefore, the focal length is adjusted so that desired optical information can be obtained according to the situation. For example, the focal length is adjusted by displacing a zoom lens included in the imaging optical system. Also, the focal length is adjusted by switching between a plurality of fixed focal length lenses (see Patent Documents 1 and 2). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Japanese Patent Application Publication No. 11-311832 [Patent Document 2] JP 2004-279556 A Summary of the Invention [Problem to be solved by the invention]
[0005] In Patent Documents 1 and 2, the expanded optical information and the wide-range optical information can be provided separately by switching the focal length, but the wide-range and expanded optical information cannot be obtained simultaneously.
[0006] In view of the above, an object of the present disclosure is to provide an imaging device that generates optical information over a wide range and in an expanded range. [Means for solving the problem]
[0007] An image processing device according to a first aspect of the present invention comprises: an optical system that forms an image of the incident first light on a predetermined light receiving area of an image sensor; a reflection optical element that reflects a second light beam that is incident on the optical system, the second light beam having an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light beam, guides the second light beam to the predetermined light receiving area, and superimposes the second light beam on an image formed by the first light beam and causes the image sensor to capture the image, a controller that separates an image output by the image sensor, in which an image based on the first light and an image based on the second light are superimposed, into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor, and a second image component corresponding to an object point outside the direct angle of view; The controller combines a mirror-inverted image of the second image component with an image based on the first image component.
[0008] An imaging apparatus according to a second aspect comprises: An imaging element; an optical system that forms an image of the incident first light on a predetermined light receiving area of the image sensor; a reflecting optical element that reflects a second light beam that is incident on the optical system, the second light beam having an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light beam, guides the second light beam to the predetermined light receiving area, and superimposes the second light beam on an image formed by the first light beam to be captured by the image sensor; and a controller that separates an image output by the image sensor, in which an image based on the first light and an image based on the second light are superimposed, into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor, and a second image component corresponding to an object point outside the direct angle of view; The controller combines a mirror-inverted image of the second image component with an image based on the first image component.
[0009] An image processing method according to a third aspect comprises: an optical system that forms an image of the incident first light on a predetermined light receiving area of an image sensor; a reflection optical element that reflects a second light beam that is incident on the optical system, the second light beam having an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light beam, guides the second light beam to the predetermined light receiving area, and superimposes the second light beam on an image formed by the first light beam and causes the image sensor to capture the image, acquiring an image in which the image generated by the first light and the image generated by the second light are superimposed, the image being output by the image sensor; Separating the image into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor and a second image component corresponding to an object point outside the direct angle of view; A horizontally inverted image of the second image component and an image based on the first image component are combined. Effect of the Invention
[0010] According to the present disclosure, a wide range of expanded optical information can be generated. [Brief description of the drawings]
[0011] [Figure 1] 1 is a diagram showing a schematic configuration of an imaging device according to a first embodiment. [Diagram 2] 2 is a diagram showing a modification of the optical element in FIG. 1, as viewed from a direction perpendicular to the optical axis of the imaging device. [Diagram 3] 1. FIG. 2 is a diagram showing an imaging device viewed from a direction perpendicular to the optical axis, in order to show another modified example of the optical element in FIG. [Figure 4] 1. FIG. 2 is a diagram showing an imaging device viewed from a direction perpendicular to the optical axis, in order to show yet another modified example of the optical element in FIG. [Diagram 5] 1. FIG. 2 is a diagram showing an imaging device viewed from a direction perpendicular to the optical axis, in order to show yet another modified example of the optical element in FIG. [Figure 6] 1. FIG. 2 is a diagram showing an imaging device viewed from a direction perpendicular to the optical axis, in order to show yet another modified example of the optical element in FIG. [Figure 7]1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 8] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 9] 2 is a diagram for explaining the physical properties of the imaging element and the optical system in FIG. [Figure 10] 2 is a conceptual diagram for explaining an image that reaches the light receiving region of FIG. 1. [Figure 11] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 12] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 13] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 14] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 15] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 16] 1. FIG. 4 is a diagram showing an imaging device viewed from the normal direction of a light receiving region, in order to show still another modified example of the optical element in FIG. [Figure 17] 2 is a conceptual diagram for explaining a situation in which a superimposed image reaching the light receiving region in FIG. 1 is formed. [Figure 18] 2 is a conceptual diagram for explaining a process of generating a restored image from a superimposed image by the controller of FIG. 1. FIG. [Figure 19] 4 is a flowchart for explaining a distance measurement process executed by the controller of FIG. 1. [Figure 20] FIG. 11 is a diagram showing a schematic configuration of an imaging device according to a second embodiment. [Figure 21] 21 is a conceptual diagram for explaining image components that reach the light receiving region of FIG. 20. [Figure 22]21 is a conceptual diagram for explaining image components that reach a light receiving region in the modified example of FIG. 20. [Diagram 23] 21 is a conceptual diagram for explaining image components that reach a light receiving region in another modified example of FIG. 20. FIG. [Figure 24] FIG. 11 is a diagram showing a schematic configuration of an imaging device according to a third embodiment. [Diagram 25] 25 is a configuration diagram for explaining a pixel structure in the image sensor of FIG. 24. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In the following drawings, the same components are denoted by the same reference numerals.
[0013] 1, an imaging device 10 including an optical device 21 according to a first embodiment of the present disclosure includes the optical device 21 and an imaging element 12. The imaging device 10 may further include a controller 14. The optical device 21 includes an imaging optical system (optical system) 11 and an optical element 13.
[0014] The imaging optical system 11 forms an image of the incident subject light beam. The imaging optical system 11 forms an image of the incident first light in a predetermined area pa. The first light may be light emitted from an object point located within the angle of view of the imaging optical system 11 alone. The predetermined area pa may be, for example, a virtual plane or curved surface in a three-dimensional space whose center intersects with the optical axis ox of the imaging optical system 11. Hereinafter, the angle of view of the imaging optical system 11 alone, in other words, the angle of view of the imaging optical system 11 in a configuration that does not include the optical element 13, is also called the direct angle of view. The imaging optical system 11 is composed of an optical element that forms an image of light beams emitted from object points at different positions at different image points by itself, in other words, without the optical element 13. The optical elements that constitute the imaging optical system 11 are, for example, lenses, mirrors, diaphragms, etc.
[0015] The imaging optical system 11 does not have to be image-side telecentric. In other words, the angle of the chief ray of any light beam passing through the imaging optical system 11 with respect to the optical axis may be greater than 0°. Alternatively, the imaging optical system 11 may be image-side telecentric.
[0016] The optical element 13 guides the second light incident on the imaging optical system 11 to a predetermined area pa. The second light is different from the first light in the angle between the optical axis ox of the imaging optical system 11 and the chief ray incident on the imaging optical system 11. The second light may be light emitted from an object point located outside the angle of view of the imaging optical system 11, in other words, directly outside the angle of view. Therefore, the angle between the chief ray of the second light and the optical axis ox may be larger than the angle between the chief ray of the first light and the optical axis ox. The chief ray may be any of a ray passing through the center of the aperture stop of the imaging optical system 15, a ray passing through the center of the entrance pupil of the imaging optical system 15, and a ray at the center of a light beam emitted from an arbitrary object point and incident on the imaging optical system 15. Furthermore, the optical element 13 may image the second light passing through the imaging optical system 11 in a predetermined area pa.
[0017] The optical element 13 may be a mirror that reflects the second light and guides it to a predetermined area pa. The reflective surface of the mirror may be parallel to the optical axis ox of the imaging optical system 11. Alternatively, the reflective surface of the mirror does not have to be parallel to the optical axis ox.
[0018] 2, the reflective surface of the mirror may be tilted with respect to the optical axis ox so as to assume an outwardly tilted position facing the imaging optical system 11. In the outwardly tilted position, the angle of view of the entire optical device 21 can be made wider than in a configuration in which the reflective surface of the mirror is parallel to the optical axis ox.
[0019] In the outwardly inclined configuration, the optical device 21 may include a first lens 22 for adjusting the optical path length between the imaging optical system 11 and the optical element 13, which is a mirror. By providing the first lens 22, the deviation of the focal position from the predetermined area pa due to the optical path length being longer than in a configuration in which the reflecting surface of the mirror is parallel to the optical axis ox can be reduced. The first lens 22 may be a cylindrical lens in a configuration in which the mirror is a mirror having a surface parallel to the direction perpendicular to the optical axis ox, such as a flat mirror. The first lens 22 may be located outside the optical axis ox of a line connecting the principal ray of the first light passing through the outer edge of the exit pupil of the imaging optical system 11 and the predetermined area pa.
[0020] 3, the optical device 21 may be provided with a prism 23. The second light may be reflected by the optical element 13, which is a mirror, and further reflected by the prism 23 to be guided to a predetermined area pa. By providing the prism 23, the inclination angle between the mirror and the optical axis ox in the outwardly inclined configuration can be made wider.
[0021] In the outward tilt configuration, the mirror optical element 13 may be, for example, a flat mirror, a curved mirror, a DMD, and a Fresnel mirror.
[0022] 4, the reflective surface of the mirror may be tilted with respect to the optical axis ox so as to assume an inwardly tilted position facing the image forming surface of the imaging optical system 11. In the inwardly tilted position, the entire optical device 21 can be made smaller than in a configuration in which the reflective surface of the mirror is parallel to the optical axis ox.
[0023] In the inward tilt configuration, the optical element 13 which is a mirror may be, for example, a flat mirror, a curved mirror as shown in FIG. 5, a DMD as shown in FIG. 6, and a Fresnel mirror.
[0024] The reflective surface of the mirror may be parallel to any side of the rectangular light receiving area ra of the image sensor 12 described later. Alternatively, the reflective surface of the mirror may intersect any side of the light receiving area ra as shown in FIG. 7. In a configuration in which the reflective surface of the mirror intersects any side of the light receiving area ra, the separation accuracy by the image separation model described later can be improved. In a configuration in which the reflective surface of the mirror intersects any side of the light receiving area ra, it is preferable to arrange the optical element 13 so that the overlap area between the area sandwiched between two straight lines extending perpendicularly from both ends of the optical element 13, which is a mirror, and the light receiving area ra is maximized when viewed from the normal direction of the light receiving area ra.
[0025] The mirror may be located outside the exit pupil of the imaging optical system 11 as viewed from the optical axis ox direction of the imaging optical system 11. More specifically, the mirror may be disposed with respect to the imaging optical system 11 such that the reflecting surface is located outside the exit pupil. Alternatively, the mirror may be located inside the exit pupil as viewed from the optical axis ox direction. In particular, in a configuration in which the light-receiving area ra is smaller than the pupil diameter, the mirror may be located inside the exit pupil.
[0026] The mirror may include a plurality of plane mirrors. Two plane mirrors belonging to at least one set of the plurality of plane mirrors may be positioned so that the reflective surfaces are opposite and parallel to each other. Alternatively, the plurality of plane mirrors may be two plane mirrors positioned so that the reflective surfaces are perpendicular to each other, as shown in FIG. 8. Furthermore, the two plane mirrors whose reflective surfaces are perpendicular to each other may be parallel to two perpendicular sides of the rectangular light receiving area ra. The plane mirror and the outer edge of the light receiving area ra of the image sensor 12 may be in close contact with each other in the normal direction of the plane mirror. Alternatively, the plane mirror and the outer edge of the light receiving area ra may not be in close contact with each other in the normal direction of the plane mirror, but may have a gap therebetween.
[0027] 9, the distance H between the optical axis ox and each of the two plane mirrors whose reflective surfaces are parallel to each other may be equal. The two parallel plane mirrors, the imaging optical system 11, and the imaging element 12 satisfy CRA≦tan -1The imaging optical system 11 may be designed and arranged to satisfy (H / B). CRA is the angle of the principal ray of the light beam emitted from the object point pp at an angle twice the direct angle of view with respect to the optical axis ox of the imaging optical system 11. B is the back focus of the imaging optical system 11.
[0028] As will be described later, by combining the arrangement of the image sensor 12 in the imaging device 10 and the above-mentioned configuration, as shown in Fig. 10, a first image component im1 corresponding to an object point within the direct angle of view, in other words, an object point emitting a first light, reaches the light receiving area ra of the image sensor 12 without passing through the optical element 13. More specifically, the first image component im1 corresponding to an object point within the direct angle of view corresponds to an object image located within the direct angle of view. Also, a second image component im2 corresponding to an object point outside the direct angle of view, in other words, an object point emitting a second light, reaches the light receiving area ra in an inverted form via the optical element 13. More specifically, the second image component im2 corresponding to an object point outside the direct angle of view corresponds to an object image located outside the direct angle of view.
[0029] In the above description, the optical element 13 is a mirror having a surface parallel to the direction perpendicular to the optical axis ox, but it may be a mirror having a curved surface as viewed from the optical axis ox. For example, the optical element 13 may be a pair of curved mirrors provided on a pair of opposite sides of a rectangular light receiving region ra as viewed from the normal direction of the light receiving region ra, as shown in FIG. 11. The curved mirrors may be parallel to the normal direction of the light receiving region ra. Alternatively, the optical element 13 may be a mirror having a circular curved surface that includes the rectangular light receiving region ra as viewed from the normal direction of the light receiving region ra, as shown in FIG. 12. Alternatively, the optical element 13 may be a mirror having an elliptical curved surface that includes the rectangular light receiving region ra as viewed from the normal direction of the light receiving region ra, as shown in FIG. 13. A mirror having an elliptical curved surface is preferable in a configuration in which the light receiving region ra is a rectangle other than a square. Alternatively, the optical element 13 may be a mirror having a circular curved surface that is included in the rectangular light receiving region ra when viewed from the normal direction of the light receiving region ra, as shown in Fig. 14. Alternatively, the optical element 13 may be a mirror having an elliptical curved surface that is included in the rectangular light receiving region ra when viewed from the normal direction of the light receiving region ra, as shown in Fig. 15. In a configuration in which the optical element 13 is a mirror having a curved surface that is included in the rectangular light receiving region ra, a gap between the light receiving region ra and the mirror can be eliminated when viewed from the normal direction of the light receiving region ra. In such a configuration, the elimination of the gap can improve the continuity of optical information in the superimposed image described below, compared to a configuration in which a gap is included.
[0030] The imaging element 12 captures an image formed within a light receiving area ra. The imaging element 12 may be disposed in the imaging device 10 such that a predetermined area pa of the optical device 21 and the light receiving area ra overlap. Therefore, the light receiving area ra of the imaging element 12 may correspond to a direct angle of view. The direct angle of view may be an angle of view corresponding to a range of object points imaged within the light receiving area ra without passing through the optical element 13. At least a part of a light flux, which is a first light, entering the imaging optical system 11 from within the direct angle of view of the imaging optical system 11 may be imaged in the light receiving area ra. Also, at least a part of a light flux, which is a second light, entering the imaging optical system 11 from outside the direct angle of view of the imaging optical system 11 and passing through the optical element 13 may be imaged in the light receiving area ra.
[0031] The imaging element 12 may be capable of capturing an image using invisible light such as visible light, infrared light, and ultraviolet light. The imaging element 12 may be, for example, a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like. The imaging element 12 may be a color image sensor. In other words, a plurality of pixels arranged in a light receiving area ra of the imaging element 12 may be covered by, for example, RGB color filters so as to be evenly distributed within the light receiving area ra. The imaging element 12 generates an image signal corresponding to an image received by imaging. The imaging element 12 may generate the image signal at a predetermined frame rate such as 30 fps.
[0032] In the image sensor 12, the outer edge of the light receiving region ra on the side where the optical element 13 is provided may be located outside the outer edge of the exit pupil of the image capturing optical system 11. Outside the outer edge of the exit pupil means outside with respect to the optical axis ox of the image capturing optical system 11. As described above, the light receiving region ra may be rectangular.
[0033] The imaging device 10 may be provided with a plurality of imaging elements 12. In a configuration in which a plurality of imaging elements 12 are provided, an optical element 13 may be provided between two adjacent imaging elements 12, as shown in Fig. 16. By providing the optical element 13 between the two adjacent imaging elements 12, a subject light flux that is imaged in a gap that occurs between the light receiving regions ra of the two adjacent imaging elements 12 in a configuration in which the optical element 13 is not provided can be imaged by at least one of the imaging elements 12.
[0034] In addition, with the above-mentioned configuration, the first image component im1 and the second image component im2 inverted when the optical element 13 is a mirror are superimposed in the light receiving area ra as shown in Fig. 17. Therefore, the imaging element 12 captures a superimposed image olim of the first image component im1 and the second image component im2 inverted when the optical element 13 is a mirror.
[0035] The controller 14 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a central processing unit (CPU) or a graphics processing unit (GPU), or a dedicated processor specialized for a specific process. The dedicated circuit may be, for example, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. The controller 14 may perform image processing on an image signal acquired from the imaging element 12.
[0036] As shown in FIG. 18, the controller 14 may perform image processing to separate the superimposed image olim corresponding to the image signal into a first image component im1 and a second image component im2 by image processing. The controller 14 separates the superimposed image olim by applying an image processing method such as an independent component analysis, a wavelet method, an image separation model, etc. The image separation model is a model constructed by, for example, creating a superimposed image in which a plurality of images are superimposed in advance, and learning the plurality of images as correct answers for the superimposed image. The image separation model may be a model that applies Pix-to-Pix, which makes a generator that generates an image, like an Encoder-Decoder model, compete with a discriminator that determines whether the generated image is a fake image, and generates a pair image that reflects the relationship between them. The controller 14 may generate a restored image rcim by combining the separated first image component im1 and second image component im2.
[0037] The controller 14 may use the restored image rcim to measure the distance to a subject captured around the imaging device 10. The controller 14 may use the restored image rcim to measure the distance based on, for example, Depth From Defocus (DFD). The controller 14 may use the restored image rcim to measure the distance based on a motion parallax method (Simultaneous Localization and Mapping, Motion Stereo (SLAM)), a separation model based on Deep Learning, a foot distance measurement method, or the like. The foot distance measurement method is a method of calculating three-dimensional coordinates based on image coordinates on the assumption that the bottom end of the subject image is located on the ground.
[0038] The controller 14 may generate a distance image based on the distance corresponding to each address of the restored image rcim. The distance image is an image in which the pixel value of each pixel corresponds to the distance. The controller 14 may provide the distance image to an external device.
[0039] Next, the distance measurement process executed by the controller 14 in this embodiment will be described with reference to the flowchart of Fig. 19. The distance measurement process starts every time an image signal is acquired from the imaging element 12.
[0040] In step S100, the controller 14 separates the second image component im2 from the overlapped image olim corresponding to the acquired image signal. After separation, the process proceeds to step S101.
[0041] In step S101, the controller 14 generates a first image component im1 by subtracting the second image component im2 separated in step S100 from the overlapped image olim. After the first image component im1 is generated, the process proceeds to step S102.
[0042] In step S102, the controller 14 generates a restored image rcim by combining the second image component im2 separated in step S101 and the first image component im1 generated in step S101. After generating the restored image rcim, the process proceeds to step S103.
[0043] In step S103, the controller 14 measures the distance to each object appearing in the restored image rcim using the restored image rcim generated in step S102. After measuring the distance, the process proceeds to step S104.
[0044] In step S104, the controller 14 generates a distance image based on the distance calculated in step S103 and the position of the restored image rcim corresponding to the distance. The controller 14 also provides the distance image to the external device. After the distance image is generated, the distance measurement process ends.
[0045] The optical device 21 of the first embodiment configured as described above includes the imaging optical system 11 that forms an image of the incident first light in a predetermined area pa, and the optical element 13 that guides the second light, the angle of which between the optical axis ox of the imaging optical system 11 and the principal ray incident on the imaging optical system 11 is different from that of the first light, to the predetermined area pa. With this configuration, the optical device 21 can employ the imaging optical system 11 with a relatively long focal length and guide an image including optical information in a wider angle range than the angle of view corresponding to the focal length to the predetermined area ra. Therefore, the optical device 21 can generate wide-range and enlarged optical information.
[0046] Moreover, in the imaging device 10 of the first embodiment, the optical element 13 is a mirror that reflects the second light and guides it to a predetermined area pa, and the reflective surface of the mirror is parallel to the optical axis ox and to one of the sides of the rectangular light-receiving area ra of the imaging element 12. With this configuration, in the imaging device 10, the image that reaches the light-receiving area ra via the mirror is distorted in one direction or less. Therefore, the imaging device 10 can reduce the load of image processing that removes distortion of the reflected light component caused by the mirror included in the captured image, and can improve the reproducibility of the reflected light component.
[0047] In the imaging device 10 of the first embodiment, the mirror includes a plurality of plane mirrors, and at least one pair of the plurality of plane mirrors is positioned so that the reflecting surfaces of the two plane mirrors are parallel to each other. With this configuration, the imaging device 10 can obtain optical information expanded on both sides from the direct angle of view centered on the optical axis ox.
[0048] In the imaging device 10 of the first embodiment, the distance H between the optical axis ox and each of the reflecting surfaces of the two parallel plane mirrors is equal, and the angle CRA of the chief ray of the light beam from the object point pp at an angle twice the direct angle of view satisfies CRA≦tan -1 With this configuration, the imaging device 10 can prevent overlapping of three layers of image components caused by overlapping of the light components reflected from one plane mirror with the light components reflected from the other plane mirror, thereby improving the accuracy of separation of the image components in the subsequent image processing.
[0049] In the imaging device 10 of the first embodiment, the plane mirror and the outer edge of the light receiving area ra of the image sensor 12 are in close contact with each other in the normal direction of the plane mirror. If a gap occurs between the plane mirror and the light receiving area ra of the image sensor 12 in the normal direction, optical information of the subject imaged in the gap is lost. In response to such an event, the imaging device 10 having the above-mentioned configuration prevents the loss of optical information.
[0050] In the imaging device 10 of the first embodiment, the mirror is located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. With this configuration, the imaging device 10 can cause a light beam passing through the vicinity of the edge of the exit pupil to be incident on the mirror. Therefore, the imaging device 10 can reduce a decrease in the amount of light caused by vignetting of a part of the light beam passing through the vicinity of the exit pupil.
[0051] In the imaging device 10 of the first embodiment, the angle of the chief ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. With this configuration, in the imaging device 10, the imaging optical system 11 is not image-side telecentric, so that a light beam from an object point at an angle wider than the direct angle of view can be made incident on the optical element 13. Therefore, the imaging device 10 can reliably generate optical information in a range that is wider than the direct angle of view.
[0052] The imaging device 10 of the first embodiment also includes a controller 14 that separates an image corresponding to an image signal into a first image component im1 directly corresponding to an object point in the image and a second image component im2 directly corresponding to an object point outside the angle of view. With this configuration, the imaging device 10 can generate an image in which the superimposition of a superimposed image olim, in which multiple image components are superimposed, is eliminated.
[0053] Next, an imaging device according to a second embodiment of the present disclosure will be described. In the second embodiment, the configuration of the optical element and the separation process by the controller are different from those of the first embodiment. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that the same reference numerals are used to designate parts having the same configuration as the first embodiment.
[0054] 20, the imaging device 100 according to the second embodiment is similar to the first embodiment and includes an imaging optical system 11, an imaging element 12, and an optical element 130. The imaging device 100 may further include a controller 14. The structures and functions of the imaging optical system 11 and the imaging element 12 in the second embodiment are the same as those in the first embodiment. The structure of the controller 14 in the second embodiment is the same as that in the first embodiment.
[0055] In the second embodiment, similar to the first embodiment, the optical element 130 focuses at least a part of the light beam incident on the imaging optical system 11 from outside the direct angle of view of the imaging optical system 11 within the light receiving area ra of the imaging element 12. In the second embodiment, unlike the first embodiment, the optical element 130 performs optical processing on the incident light beam before outputting it.
[0056] The optical processing is, for example, a change in the band of the incident light beam. Specifically, the optical element 130 attenuates light in a band that corresponds to some of the color filters among the multiple color filters that cover the image sensor 12, among the incident light beam. Therefore, the optical element 130 forms an image of light in a band excluding the light in the attenuated band within the light receiving region ra.
[0057] The optical element 130 may be a mirror that reflects light in a band other than the color band to be attenuated. For example, the optical element 130 attenuates R light and reflects GB light.
[0058] 21, in the second embodiment, with the above-mentioned configuration, the first R image component im1r, the first G image component im1g, and the first B image component im1b, which correspond to object points within the direct angle of view and correspond to the colors of all color filters, reach the light-receiving region ra without passing through the optical element 130. Also, the second G image component im2g and the second B image component im2b, which correspond to object points outside the direct angle of view and correspond to colors other than the attenuated color components, reach the light-receiving region ra via the optical element 130.
[0059] Alternatively, the optical processing is, for example, the addition of a contrast pattern to the incident light beam according to the incident position. Specifically, as shown in FIG. 22, the optical element 130 has a surface in which the first region 190 and the second region 200 are distributed in a checkered pattern, for example. The first region 190 attenuates the luminance of the incident light at a first attenuation rate and outputs the light. The first attenuation rate is greater than 0% and less than 100%. The second region 200 attenuates the luminance of the incident light at a second attenuation rate and outputs the light. The second attenuation rate is greater than 0% and less than the first attenuation rate. Therefore, the optical element 130 imparts a contrast pattern to the incident light beam according to the patterns of the first region 190 and the second region 200, and forms an image in the light receiving region ra.
[0060] Alternatively, the optical processing is, for example, the addition of distortion to an image formed in the light receiving region ra by the incident light beam. Specifically, as shown in Fig. 23, the optical element 130 is a mirror having a cylindrical curved surface with an axis parallel to the optical axis, and reflects the incident light beam to form a distorted image in the light receiving region ra. More specifically, the optical element 130 is an image distorted so as to expand in a direction connecting both ends of the arc of the cross section of the optical element, which is a mirror with a plane perpendicular to the optical axis.
[0061] In the second embodiment, the controller 14 may perform image processing to separate the superimposed image olim corresponding to the image signal into a first image component im1 and a second image component im2, similar to the first embodiment. In the second embodiment, the controller 14 separates the superimposed image olim by an image processing method using an image separation model.
[0062] The image separation model in the second embodiment will be described below. The image separation model is constructed by generating a superimposed image in advance, which is a first image not subjected to optical processing and a second image different from the first image, superimposed on the first image by performing image processing equivalent to the optical processing performed by the optical element 130, and learning the first image and the second image as correct answers for the superimposed image.
[0063] In a configuration in which the optical processing is a band change, the first image is the RGB image components of an arbitrary image, and in the same configuration, the second image is the GB image components of an image other than the arbitrary image. In a configuration in which the optical processing is a band change, the R image component of the arbitrary image may also be used for learning in addition to the superimposed image.
[0064] In a configuration in which the optical processing is the addition of a contrast pattern, the first image is an arbitrary image. In the configuration, the second image is an image obtained by varying the luminance of an image other than the arbitrary image using the contrast pattern of the optical element 130.
[0065] In a configuration in which the optical processing is the application of distortion, the first image is an arbitrary image. In this configuration, the second image is an image other than the arbitrary image, which is reflected by a mirror having the same curved surface as the optical element 130.
[0066] In the second embodiment, the controller 14 may generate a restored image rcim by combining the separated first image component im1 and second image component im2, as in the first embodiment. In the second embodiment, the controller 14 may use the restored image rcim to measure the distance to a subject captured in the vicinity of the imaging device 100, as in the first embodiment. In the second embodiment, the controller 14 may generate a distance image based on the distance corresponding to each address of the restored image rcim, and provide the image to an external device, as in the first embodiment.
[0067] The optical device 210 of the second embodiment configured as described above also includes the imaging optical system 11 that forms an image of the incident first light in a predetermined area pa, and the optical element 130 that guides to the predetermined area pa the second light, the second light having an angle between the optical axis ox of the imaging optical system 11 and the principal ray incident on the imaging optical system 11 that is different from that of the first light. Therefore, the imaging device 100 can also generate optical information that is wide-ranged and expanded.
[0068] Also, in the imaging device 100 of the second embodiment, the optical element 130 is a mirror that reflects the second light to form an image in a predetermined area pa, and the reflective surface of the mirror is parallel to the optical axis ox and any side of the rectangular light receiving area ra of the imaging element 12. Therefore, the imaging device 100 can also reduce the load of image processing that removes distortion of the reflected light component by the mirror contained in the captured image, thereby improving the reproducibility of the reflected light component.
[0069] In the imaging device 100 of the second embodiment, the mirrors include a plurality of plane mirrors, and at least one pair of the plane mirrors is positioned so that the reflecting surfaces are parallel to each other. Therefore, the imaging device 100 can also obtain optical information expanded on both sides from the direct angle of view centered on the optical axis ox.
[0070] In the imaging device 100 of the second embodiment, the distance H between the optical axis ox and each of the reflecting surfaces of the two parallel plane mirrors is equal, and the angle CRA of the chief ray of the light beam from the object point pp at an angle twice the direct angle of view satisfies CRA≦tan -1 Therefore, in the imaging device 100, the overlap of three layers of image components caused by the overlap of the reflected light component from one plane mirror and the reflected light component from the other plane mirror is prevented, and the accuracy of separation of the image components in the subsequent image processing can be improved.
[0071] Also, in the imaging device 100 of the second embodiment, the plane mirror and the outer edge of the light receiving area ra of the imaging element 12 are in close contact with each other in the normal direction of the plane mirror. Therefore, the imaging device 100 also prevents loss of optical information.
[0072] Also, in the imaging device 100 of the second embodiment, the mirror is located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. Therefore, the imaging device 100 can also reduce a reduction in the amount of light caused by vignetting of a part of the light flux passing through the vicinity of the exit pupil.
[0073] Also, in the imaging device 100 of the second embodiment, the angle of the chief ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. Therefore, the imaging device 100 can also reliably generate optical information in a range that is wider than the direct angle of view.
[0074] The imaging device 100 of the second embodiment also includes a controller 14 that separates an image corresponding to an image signal into a first image component im1 directly corresponding to an object point in the image and a second image component im2 directly corresponding to an object point outside the angle of view. Therefore, the imaging device 100 can also generate an image in which the superimposition of a superimposed image olim in which a plurality of image components are superimposed is eliminated.
[0075] In the imaging device 100 of the second embodiment, the optical element 130 performs optical processing on the light beam incident on the optical element 130 and outputs the processed light. With this configuration, the imaging device 100 imparts optical characteristics corresponding to the optical processing to the separated image components. Therefore, the imaging device 100 can construct an image separation model that has been trained to improve separation accuracy. Therefore, the imaging device 100 can improve the restoration accuracy of the restored image.
[0076] Next, an imaging device according to a third embodiment of the present disclosure will be described. In the third embodiment, the configuration of the imaging element and the separation process by the controller are different from those of the first embodiment. The third embodiment will be described below, focusing on the differences from the first embodiment. Note that the same reference numerals are used to designate parts having the same configuration as those of the first embodiment.
[0077] 24, an imaging device 101 according to the third embodiment is similar to the first embodiment and includes an imaging optical system 11, an imaging element 121, and an optical element 13. The imaging device 101 may further include a controller 14. The structures and functions of the imaging optical system 11 and the optical element 13 in the third embodiment are the same as those in the first embodiment. The structure of the controller 14 in the third embodiment is the same as that in the first embodiment.
[0078] In the third embodiment, the image sensor 121 captures an image formed in the light receiving region ra through the imaging optical system 11, similar to the first embodiment. The image sensor 121 may be capable of capturing an image by invisible light such as visible light, infrared light, and ultraviolet light, similar to the first embodiment. The image sensor 121 may be a color image sensor. The image sensor 121 generates an image signal corresponding to an image received by imaging, similar to the first embodiment. The image sensor 121 may generate an image signal at a predetermined frame rate such as 30 fps, similar to the first embodiment. In the image sensor 121, the outer edge of the light receiving region ra on the side where the optical element 13 is provided may be located outside the outer edge of the exit pupil of the imaging optical system 11, similar to the first embodiment. The light receiving region ra may be rectangular, similar to the first embodiment.
[0079] In the third embodiment, the image sensor 121 may be a dual pixel type image sensor, unlike the first embodiment. As shown in Fig. 25, the image sensor 121, which is a dual pixel type image sensor, is an image sensor having a structure in which a first PD (PhotoDiode) 171 and a second PD 181 are provided in the pixels 161 covered by each microlens 151, and a light beam can be incident only on one of the PDs depending on the incident direction of the light beam. For example, in each pixel 161, only a light beam from a direction inclined toward the optical axis ox side can be incident on the first PD 171, and only a light beam from a direction inclined toward the optical element 13 side can be incident on the second PD 181.
[0080] With the above-mentioned configuration, the first image component im1 corresponding to an object point in the direct image reaches the first PD 171 in the light-receiving region ra without passing through the optical element 13. Also, the second image component im2 corresponding to an object point outside the direct angle of view reaches the second PD 181 in the light-receiving region ra in an inverted form via the optical element 13.
[0081] In the third embodiment, the controller 14 may perform image processing to separate the superimposed image olim corresponding to the image signal into a first image component im1 and a second image component im2, similar to the first embodiment. In the third embodiment, the controller 14 may generate the first image component im1 based only on the signal generated by the first PD171. Based only on the signal generated by the first PD171 specifies that the signal generated by the second PD181 is not used, and may include using a signal unrelated to the signal output by the second PD181, such as a synchronization signal. In the third embodiment, the controller 14 may generate the second image component im2 inverted based only on the signal generated by the second PD181. Based only on the signal generated by the second PD181 has a similar meaning to based only on the signal generated by the first PD171.
[0082] In the third embodiment, the controller 14 may generate a restored image rcim by combining the separated first image component im1 and second image component im2, as in the first embodiment. In the third embodiment, the controller 14 may use the restored image rcim to measure the distance to a subject captured in the vicinity of the imaging device 101, as in the first embodiment. In the third embodiment, the controller 14 may generate a distance image based on the distance corresponding to each address of the restored image rcim, and provide the image to an external device, as in the first embodiment.
[0083] In the imaging device 101 of the third embodiment configured as described above, the optical element 13 is also a mirror that reflects the second light and guides it into a predetermined area pa, and the reflective surface of the mirror is parallel to the optical axis ox and any side of the rectangular light-receiving area ra of the imaging element 121. Therefore, the imaging device 101 can also reduce the load of image processing that removes distortion of the reflected light component by the mirror contained in the captured image, thereby improving the reproducibility of the reflected light component.
[0084] In the imaging device 101 of the third embodiment, the mirrors also include a plurality of plane mirrors, and at least one pair of the plurality of plane mirrors is positioned so that the reflecting surfaces are parallel to each other. Therefore, the imaging device 101 can also acquire optical information expanded on both sides from the direct angle of view centered on the optical axis ox.
[0085] In the imaging device 101 of the third embodiment, the distance H between the optical axis ox and each of the reflecting surfaces of the two parallel plane mirrors is equal, and the angle CRA of the chief ray of the light beam from the object point pp at an angle twice the direct angle of view satisfies CRA≦tan -1 Therefore, in the imaging device 101, the overlap of three layers of image components caused by the overlap of the reflected light components from one plane mirror and the reflected light components from the other plane mirror can be prevented, and the accuracy of separation of the image components in the subsequent image processing can be improved.
[0086] Also, in the imaging device 101 of the third embodiment, the plane mirror and the outer edge of the light receiving area ra of the imaging element 121 are in close contact with each other in the normal direction of the plane mirror. Therefore, the imaging device 101 also prevents loss of optical information.
[0087] Also, in the imaging device 101 of the third embodiment, the mirror is located outside the exit pupil of the imaging optical system 11 when viewed from the optical axis ox direction. Therefore, the imaging device 101 can also reduce a reduction in the amount of light caused by vignetting of a part of the light flux passing through the vicinity of the exit pupil.
[0088] Also, in the imaging device 101 of the third embodiment, the angle of the chief ray of any light beam in the imaging optical system 11 with respect to the optical axis ox is greater than 0°. Therefore, the imaging device 101 can also reliably generate optical information in a range that is wider than the direct angle of view.
[0089] The imaging device 101 of the third embodiment also includes a controller 14 that separates an image corresponding to an image signal into a first image component im1 directly corresponding to an object point in the image and a second image component im2 directly corresponding to an object point outside the angle of view. Therefore, the imaging device 101 can also generate an image in which the superimposition of a superimposed image olim in which a plurality of image components are superimposed is eliminated. EXAMPLES
[0090] A driving space image was prepared as a data set of images for learning. Using each image of these data sets, a part of the whole image was cut, and a set of superimposed images in which one partial image was superimposed on the other partial image was created. An image separation model for separating superimposed images was trained using the created 15,000 sets as teacher data to construct the image separation model of Example 1. The remaining sets were used as confirmation data to separate the superimposed images using the image separation model of Example 1. The separated images were compared with the partial images used for the superimposed images to calculate the PSNR and SSIM. In the image separation model of Example 1, the PSNR and SSIM were 20.54 and 0.55, respectively.
[0091] Using each image of the same data set as in Example 1, a part of the whole image was cut off, and a set of superimposed images was created by superimposing the GB image component of one partial image on the RGB image component of the other partial image. An image separation model for separating the superimposed images was trained using the created X1 set as training data to construct an image separation model of Example 2. The remaining set was used as confirmation data to separate the superimposed images using the image separation model of Example 2. The separated image was compared with the partial image used for the superimposed image to calculate the PSNR and SSIM. In the image separation model of Example 2, the PSNR and SSIM were 27.40 and 0.92, respectively.
[0092] Using each image of the same data set as in Example 1, a part of the whole image was cut, and the luminance of one partial image was attenuated by a checkered light-dark contrast pattern of squares corresponding to a 25° angle of view, and a set of superimposed images was created by superimposing the other partial image. The attenuation rate of one area of the light-dark contrast pattern was 70%, and the attenuation rate of the other area was 30%. An image separation model for separating superimposed images was trained using the created X1 set as teacher data, and an image separation model of Example 3 was constructed. The remaining set was used as confirmation data to separate the superimposed images using the image separation model of Example 3. The separated image was compared with the partial image used for the superimposed image to calculate the PSNR and SSIM. In the image separation model of Example 3, the PSNR and SSIM were 24.48 and 0.88, respectively.
[0093] Using each image of the same data set as in Example 1, a part of the whole image was cut, one partial image was distorted so as to reproduce the distortion aberration, and a set of superimposed images was created by superimposing the other partial image. An image separation model for separating the superimposed images was trained using the created X1 set as training data to construct an image separation model of Example 4. The remaining set was used as confirmation data to separate the superimposed images using the image separation model of Example 4. The separated image was compared with the partial image used for the superimposed image to calculate the PSNR and SSIM. In the image separation model of Example 4, the PSNR and SSIM were 25.26 and 0.90, respectively.
[0094] In one embodiment, (1) an image processing device includes: an optical system that forms an image of the incident first light on a predetermined light receiving area of an image sensor; a reflection optical element that reflects a second light beam that is incident on the optical system, the second light beam having an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light beam, guides the second light beam to the predetermined light receiving area, and superimposes the second light beam on an image formed by the first light beam and causes the image sensor to capture the image, a controller that separates an image output by the image sensor, in which an image based on the first light and an image based on the second light are superimposed, into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor, and a second image component corresponding to an object point outside the direct angle of view; The controller combines a mirror-inverted image of the second image component with an image based on the first image component.
[0095] (2) In the image processing device of (1) above, The reflecting optical element performs optical processing on the light beam incident on the reflecting optical element and outputs the processed light, and the controller performs the separation based on characteristics of the second image component that has been optically processed.
[0096] (3) In the image processing device of (2) above, The optical process is the variation of the bandwidth of the incident light beam.
[0097] (4) In the image processing device of (2) above, The optical processing involves adding a pattern of brightness differences to the incident light beam according to the incident position.
[0098] (5) The image processing device according to (2) above, The optical processing involves the application of distortion to an image formed within the light receiving area by an incident light beam.
[0099] (6) The imaging device is An imaging element; an optical system that forms an image of the incident first light on a predetermined light receiving area of the image sensor; a reflecting optical element that reflects a second light beam that is incident on the optical system, the second light beam having an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light beam, guides the second light beam to the predetermined light receiving area, and superimposes the second light beam on an image formed by the first light beam to be captured by the image sensor; and a controller that separates an image output by the image sensor, in which an image based on the first light and an image based on the second light are superimposed, into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor, and a second image component corresponding to an object point outside the direct angle of view; The controller combines a mirror-inverted image of the second image component with an image based on the first image component.
[0100] (7) In the imaging device of (6) above, The reflecting optical element performs optical processing on the light beam incident on the reflecting optical element and outputs the processed light, and the controller performs the separation based on characteristics of the second image component that has been optically processed.
[0101] (8) In the imaging device of (7) above, The optical process is the variation of the bandwidth of the incident light beam.
[0102] (9) In the imaging device of (7) above, The optical processing involves adding a pattern of brightness differences to the incident light beam according to the incident position.
[0103] (10) In the imaging device of (7) above, The optical processing involves the application of distortion to an image formed within the light receiving area by an incident light beam.
[0104] (11) The image processing method is an optical system that forms an image of the incident first light on a predetermined light receiving area of an image sensor; a reflection optical element that reflects a second light beam that is incident on the optical system, the second light beam having an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light beam, guides the second light beam to the predetermined light receiving area, and superimposes the second light beam on an image formed by the first light beam and causes the image sensor to capture the image, acquiring an image in which the image generated by the first light and the image generated by the second light are superimposed, the image being output by the image sensor; Separating the image into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor and a second image component corresponding to an object point outside the direct angle of view; A horizontally inverted image of the second image component and an image based on the first image component are combined.
[0105] (12) In the image processing method of (11) above, The reflecting optical element performs optical processing on the light beam incident on the reflecting optical element and outputs the processed light, and the controller performs the separation based on characteristics of the second image component that has been optically processed.
[0106] (13) In the image processing method of (12) above, The optical process is the variation of the bandwidth of the incident light beam.
[0107] (14) In the image processing method of (12) above, The optical processing involves adding a pattern of brightness differences to the incident light beam according to the incident position.
[0108] (15) In the image processing method of (12) above, The optical processing involves the application of distortion to an image formed within the light receiving area by an incident light beam.
[0109] The above describes embodiments of an imaging method using the imaging devices 10, 100, and 101. However, embodiments of the present disclosure can also be embodied as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.), in addition to a method or program for implementing the device.
[0110] The form of implementation of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may be a program module incorporated into an operating system, etc. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be configured so that a part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as necessary.
[0111] The drawings for explaining the embodiments according to the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.
[0112] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions, etc. included in each component, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, etc. can be combined into one or divided.
[0113] All of the features described in this disclosure and / or all steps of any method or process disclosed herein may be combined in any combination except those combinations in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.
[0114] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.
[0115] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure may have their numbers exchanged. For example, a first image component may exchange identifiers "first" and "second" with a second image component. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. Descriptions of identifiers such as "first" and "second" in this disclosure alone should not be used to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]
[0116] 10, 100, 101 Imaging device 11 Imaging optical system 12, 121 Image sensor 13, 130 Optical elements 14 Controller 151 Microlens 161 pixels 171 The first PD (PhotoDiode) 181 Second PD 190 First Area 200 Second Area 21, 210, 211 Optical equipment 22 The First Lens 23 Prism CRA: The angle of the chief ray of the light beam emitted from an object point at an angle twice the direct angle of view with respect to the optical axis of the imaging optical system. im1 First image component im1b First B image component im1g First G image component im1r First R image component im2 Second image component im2b Second B image component im2g Second G image component olim overlay image ox optical axis pa Prescribed area rcim restored image pp Object point at twice the direct angle of view ra light receiving area
Claims
1. an optical system that forms an image of the incident first light on a predetermined light receiving area of an image sensor; a reflection optical element that reflects a second light that is incident on the optical system, the second light having the same band as the first light and an angle between the optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light, guides the second light to the predetermined light receiving area, and superimposes the second light on an image formed by the first light to be captured by the image sensor; a controller that separates an image output by the image sensor, in which an image based on the first light and an image based on the second light are superimposed, into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor, and a second image component corresponding to an object point outside the direct angle of view; The controller combines a mirror-inverted image of the second image component with an image based on the first image component. Image processing device.
2. The image processing device according to claim 1 , wherein the reflective optical element performs optical processing on the light beam incident on the reflective optical element and outputs the light beam, and the controller performs the separation based on characteristics of the second image component that has been optically processed.
3. 3. The image processing apparatus according to claim 2, wherein the optical processing is to add a pattern of brightness differences to an incident light beam according to an incident position.
4. 3. The image processing apparatus according to claim 2, wherein said optical processing is to add distortion to an image formed within said light receiving region by an incident light beam.
5. An imaging element; an optical system that forms an image of the incident first light on a predetermined light receiving area of the image sensor; a reflecting optical element that reflects a second light that is incident on the optical system, the second light having the same band as the first light and an angle between an optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light, guides the second light to the predetermined light receiving area, and superimposes the second light on an image formed by the first light to be captured by the image sensor; a controller that separates an image output by the image sensor, in which an image based on the first light and an image based on the second light are superimposed, into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor, and a second image component corresponding to an object point outside the direct angle of view; The controller combines a mirror-inverted image of the second image component with an image based on the first image component. Imaging device.
6. The imaging device according to claim 5 , wherein the reflective optical element performs optical processing on the light beam incident on the reflective optical element and outputs the light beam, and the controller performs the separation based on characteristics of the second image component that has been optically processed.
7. 7. The imaging apparatus according to claim 6, wherein the optical processing is to add a pattern of brightness differences to an incident light beam according to an incident position.
8. 7. The imaging device according to claim 6, wherein the optical processing is to add distortion to an image formed within the light receiving region by an incident light beam.
9. an optical system that forms an image of the incident first light on a predetermined light receiving area of an image sensor; a reflection optical element that reflects a second light that is incident on the optical system, the second light having the same band as the first light and an angle between the optical axis of the optical system and a principal ray that is incident on the optical system that is different from that of the first light, guides the second light to the predetermined light receiving area, and superimposes the second light on an image formed by the first light to be captured by the image sensor; acquiring an image in which the image generated by the first light and the image generated by the second light are superimposed, the image being output by the imaging element; Separating the image into a first image component corresponding to an object point within a direct angle of view of the optical system corresponding to a light receiving area of the image sensor and a second image component corresponding to an object point outside the direct angle of view; A left-right inverted image of the second image component is combined with an image based on the first image component. Image processing methods.
10. The image processing method according to claim 9 , wherein the reflective optical element performs optical processing on a light beam incident on the reflective optical element and outputs the light beam, and performs the separation based on characteristics of the second image component that has been optically processed.
11. 11. The image processing method according to claim 10, wherein the optical processing is to add a pattern of brightness differences according to an incident position to an incident light beam.
12. 11. The image processing method according to claim 10, wherein the optical processing is to add distortion to an image formed within the light receiving region by an incident light beam.
Citation Information
Patent Citations
Three-direction monitoring image pickup camera device
JP1996111798A
Electronic imaging apparatus
JP2005065064A
Detection device and mobile body
JP2016151519A
Camera module and information processing apparatus
JP2017060107A
JP279556A
