Imaging apparatus

Abstract

The purpose of the present invention is to provide a technique enabling imaging of a plurality of visual fields at different magnifications by one imaging apparatus. This imaging apparatus is for simultaneously photographing a plurality of different visual fields at different magnifications, and is characterized by including, in order from an image sensor side, one image sensor, one imaging lens, and a plurality of curved mirrors arranged off-axis with respect to the optical axis of the imaging lens so that images are formed in different areas on the image sensor through the imaging lens.

Description

Imaging device

[0001] The present invention relates to an imaging device that simultaneously captures multiple different fields of view.

[0002] By giving a single camera multiple fields of view, the following advantages can be obtained:

[0003] It can cover a wide observation range. In applications where a single camera could only cover a limited area and therefore required multiple cameras, having a single camera with multiple fields of view allows for more efficient coverage of a wider area.

[0004] Cameras with multiple fields of view can be installed compactly. This eliminates the need to install multiple cameras individually, saving space and costs. Having multiple fields of view in a single camera is particularly useful in environments with limited installation space.

[0005] Information can be collected from multiple angles. A single camera with multiple fields of view can simultaneously acquire information from different angles. This allows for more comprehensive information to be obtained. For example, in the case of surveillance cameras, having multiple fields of view allows for simultaneous acquisition of images from multiple directions, enabling a more accurate understanding of the actions and location of the monitored subject.

[0006] High-speed processing becomes more efficient. When using multiple cameras, video data from each camera needs to be processed individually. However, by having a single camera with multiple fields of view, the acquisition and processing of video data can be centralized. This enables more efficient processing and improved real-time performance.

[0007] The advantages described above make them particularly suitable for in-vehicle imaging devices that require a wide field of view, compact size, and high speed. As a result, development has been underway on in-vehicle imaging devices that have multiple fields of view from a single camera.

[0008] For example, Patent Document 1 discloses an imaging device for vehicle peripheral monitoring that simultaneously captures images of the left, right, and front three-directional fields of view using a single image sensor. The incident light from the left and right fields of view is internally reflected by a prism and then imaged on the image sensor through an imaging lens, while the incident light from the front field of view is directly imaged on the image sensor through the imaging lens without passing through the prism.

[0009] Patent Document 2 discloses a method of imaging the incident light from the left and right fields of view by reflecting it twice with a prism and then imaging it with an image sensor through an imaging lens.

[0010] However, imaging devices with multiple fields of view using a single camera are rarely actually used. This is due to the rapid progress of price reduction and performance improvement of imaging devices in recent years. With the explosive spread of imaging devices for mobile phones, the price of imaging devices has decreased and the performance has improved. Therefore, when multiple fields of view are required, it has become common to use multiple imaging devices, and the need to use imaging devices with special optical systems has decreased. This is the same situation for imaging devices for near-infrared light.

[0011] In contrast, the situation is different for imaging devices (thermal cameras) for far-infrared light that can sense heat. Since far-infrared light has a wavelength of 7 to 14 μm, which is about 20 times the wavelength of visible light, the price of an image sensor for far-infrared light is hundreds of times that of an image sensor for visible light.

[0012] Also, regarding lenses, inexpensive materials such as glass and plastic can be used for lenses for visible light, so lenses processed by molding plastic are provided at a very low cost. On the other hand, since far-infrared light is absorbed by glass and plastic, it is necessary to use expensive materials such as germanium and chalcogenide, and the lenses are also very expensive.

[0013] Therefore, for far-infrared light, an imaging device capable of simultaneously capturing a plurality of fields of view with a single unit is required. For this reason, in this specification, a far-infrared imaging device capable of simultaneously capturing a plurality of fields of view will be described in detail. However, the configuration of this imaging device is not limited to far-infrared light, and it can also be used for imaging devices that are sensitive to other wavelengths such as visible light and near-infrared light.

[0014] Here, a far-infrared imaging device for vehicle-mounted vehicle peripheral monitoring will be described.

[0015] For the realization of a safe car society, the development of advanced driver assistance systems (ADAS) and autonomous driving is in progress. In the ADAS system, it is necessary to recognize the surrounding environment such as people, objects, and road shapes around the vehicle. As "external sensors" for recognizing the external environment, the development of various types of cameras such as radar, cameras, lidars, near-infrared cameras, and far-infrared imaging devices is in progress.

[0016] Among these, the far-infrared imaging device is a night vision device, which is an imaging device that can sense heat and capture objects even in the dark. Since the far-infrared imaging device can directly detect people from their body temperature without a lighting device, highly reliable human detection is possible.

[0017] The far-infrared imaging device has already been put into practical use in automobiles and is installed in some vehicle models mainly in high-class cars (Patent Document 3).

[0018] Japanese Patent Application No. 10-260764, Japanese Patent Laid-Open No. 2003-207836, Japanese Patent Laid-Open No. 60-231193

[0019] The problem with conventional far-infrared imaging devices was that the horizontal viewing angle was narrow, at 20° to 30°, and pedestrians near the vehicle could not be detected. Here, the functions required for a vehicle-mounted imaging device system will be considered. Fig. 2 shows a panoramic image taken from the driver's seat of a vehicle in front of an intersection. The leading automobile 21 in front is captured in the center of the screen, and bicycles 22-1 to 3 and pedestrians 23 traveling near the host vehicle are captured. In the figure, the viewing angle 24 (24x18°) of a conventional vehicle-mounted thermal camera is shown by a square frame. It can be seen that the conventional far-infrared imaging device has a narrow viewing angle and cannot detect pedestrians and bicycles near the vehicle.

[0020] As is clear from Figure 2, the vertical field of view required for an in-vehicle imaging system is sufficient at around 20° if limited to the central area. The vertical field of view at the edges should be larger than that of the central area, preferably around 30°. On the other hand, in order to prevent accidents involving pedestrians and cyclists when turning left or right, it is necessary to detect pedestrians and cyclists crossing the intersection before the intersection, and the horizontal field of view should be at least ±60°, preferably ±75° or more.

[0021] Figure 3 shows a bird's-eye view of the trajectory of a vehicle turning right or left at an intersection. To detect pedestrians crossing the intersection when turning left (Figure 3(a)) and prevent accidents involving pedestrians, it is clear that a horizontal field of view of at least 120° is necessary, and more preferably, a field of view of 150° or more (particularly desirable field of view 26) is required to detect pedestrians in front of the sidewalk. With the field of view 24° of conventional far-infrared imaging cameras 24, it is difficult to detect pedestrians at the intersection when turning right or left.

[0022] From the above considerations, it can be seen that the desirable field of view 25 for an in-vehicle imaging system is approximately 20° in the vertical center and 120-150° or more in the horizontal direction. Furthermore, it is desirable that the vertical field of view widens at the edges rather than in the center. A camera with such an extremely wide horizontal field of view has not been realized to date, not only as an in-vehicle far-infrared imaging system but also as a regular in-vehicle camera. This invention was made to consider the performance truly required for a far-infrared imaging system and to realize an ideal far-infrared imaging system.

[0023] As described above, in automotive far-infrared imaging cameras that image the thermal energy radiated from materials, an imaging device with a wide field of view only in the horizontal direction has not yet been realized.

[0024] In methods using multiple plane mirrors to capture multiple fields of view with a single camera, as described in Patent Documents 1 and 2, the multiple fields of view are captured at the same magnification. In contrast, applications that require imaging multiple fields of view may require different magnifications for each field of view.

[0025] For example, in an in-vehicle imaging system, there are distant objects (pedestrians and bicycles) in the direction of travel, and relatively close objects on both sides of the vehicle. In such cases, it is desirable to capture the direction of travel and the sides with fields of view having different magnifications. The field of view capturing the direction of travel should be high magnification, while the field of view capturing the sides should be low magnification and have a wide field of view.

[0026] Another in-vehicle application requiring multiple fields of view at different magnifications is the in-car monitoring camera. In the case of in-car monitoring, it is necessary to capture the driver's face at high magnification and the condition of all occupants at low magnification.

[0027] With conventional technology, it was difficult to capture images at different magnifications for each field of view.

[0028] This invention was made to solve the above-mentioned problems, and aims to provide an inexpensive and compact imaging device that can capture multiple fields of view with a single camera using multiple curved mirrors. Furthermore, by using multiple curved mirrors with different curvatures, it aims to obtain an imaging device that can simultaneously capture multiple fields of view with different magnifications.

[0029] To solve the above-mentioned problems, the present invention provides an imaging device that simultaneously captures multiple different fields of view using a single image sensor, comprising, in order from the image side toward the object to be captured, i.e., the field of view, one image sensor, one imaging lens, and a plurality of curved mirrors arranged off-axis so as to pass through the imaging lens and form images on different regions on the image sensor, wherein at least one of the plurality of curved mirrors has a different shape from the other curved mirrors.

[0030] According to the present invention, it is possible to simultaneously image multiple fields of view with a single imaging device. Furthermore, by giving different curvatures to the multiple curved mirrors mentioned above and appropriately setting the distance between each curved mirror and the imaging lens for each mirror curvature, it becomes possible to simultaneously image multiple fields of view with different magnifications.

[0031] This invention was made to achieve the field of view performance truly required for multiple in-vehicle far-infrared imaging device systems as described above.

[0032] To achieve the aforementioned field of view with a far-infrared imaging device, an optical system was designed in which multiple curved mirrors are positioned off-axis relative to the imaging lens, and the reflected light from each curved mirror is imaged onto different areas of the image sensor by the imaging lens. Figure 1 shows the configuration of the optical system of the imaging device 10 that captures multiple different fields of view simultaneously. Figure 1(a) shows the structure viewed from the horizontal, and Figure 1(b) shows the structure viewed from above.

[0033] The imaging device 10 comprises, in order from the image side to the field of view side, one image sensor 1, one imaging lens 2, and a plurality of curved mirrors 3-1, 3-2, 3-3 that are positioned off-axis so as to pass through the imaging lens 2 and form images on different regions on the image sensor 1.

[0034] Here, image sensor 1 is a far-infrared light image sensor (thermal sensor) that is sensitive to far-infrared rays (8-14 μm wavelength). As a thermal sensor, an FPA (Focal Plane Arrays) was used, which is a two-dimensional grid of vanadium oxide elements that cause a change in resistance due to temperature changes caused by the radiant energy emitted from an object. The sensor used had 640 pixels horizontally and 480 pixels vertically (aspect ratio 4:3). In this embodiment, the curved mirrors 3-1, 3-2, and 3-3 are revolutionary paraboloid mirrors, which are mirrors with a shape that is a part cut out of a revolutionary paraboloid, which is a mirror with a shape that is obtained by rotating a parabola around the axis of symmetry.

[0035] Each rotating parabolic mirror is positioned with its parabolic axis of symmetry tilted relative to the optical axis of the imaging lens 2 (off-axis positioning). This prevents the mirror's image from being reflected on the image sensor 1. Light rays incident on the first curved mirror 3-1 in the horizontal direction form an image 5-1 formed by the mirror in the first field of view. The second curved mirror 3-2 and the third curved mirror 3-3 similarly form images in their respective fields of view. The images formed by the convex-shaped curved mirrors 3-1, 3-2, and 3-3 are all virtual images. The images 5-1 and 5-2 formed by each curved mirror are focused by the imaging lens 2 and projected onto the image sensor 1.

[0036] Here, we define the point of interest in the field of view. The point of interest in the field of view is the point in each of the multiple fields of view formed on the image sensor 1 that requires the highest resolution. Normally, when using a single curved mirror, the resolution is highest at one point in the image, and from there, aberrations increase towards the edge of the image, degrading the resolution (coma aberration).

[0037] Therefore, it is common practice to set the field of view focus point, which provides high resolution, at the physical center of the field of view. However, depending on the application of the imaging device and the object being imaged, it may be preferable to set the field of view focus point at a location different from the physical center of the field of view. The case where the field of view focus point is set at a location different from the physical center of the field of view will be discussed later.

[0038] In Figure 1, three fields of view are realized using three curved mirrors 3-1, 3-2, and 3-3. The directions of the points of interest in each field of view, 4-4, 4-5, and 4-6, are indicated by arrows. Light rays from the first point of interest in the field of view, 4-4, are reflected by the first curved mirror 3-1 and formed on the image sensor 1 by the imaging lens 2. From the center point of the imaging lens: C (2-1) to the point of interest on the first mirror: M 1　 The mirrored image 6-1 of the first point of interest in the field of view lies on the straight line passing through (4-1). The mirrored images 5-1 and 5-2 are virtual images, and actual light rays are reflected by the mirror and therefore do not pass through the images.

[0039] Figure 1 shows three fields of view (FoV) formed by three curved mirrors.１ , FoV ２ , FoV ３ is shown by.

[0040] The imaging device 10 shown in FIG. 1 was mounted in front of the roof of a vehicle and an actual on-road imaging test was conducted. The results are shown in FIG. 4. FIG. 4(a) is a two-dimensional image obtained by the image sensor 1 (thermal image sensor). The FoV in FIG. 1 １ , FoV ２ , FoV ３ of the three viewing angles correspond to the FoV of the viewing areas on the image sensor shown in FIG. 4(a). Each viewing field divides the entire area of the image sensor. FoV １ , FoV ２ , FoV ３ corresponds. Each viewing field divides the entire area of the image sensor 1. FoV １ forms an image on the lower half of the image sensor, and the wide angles of FoV ２ , FoV ３ form images on the left and right of the upper part of the image sensor. FoV １ captures the front in the vehicle traveling direction at a high magnification. FoV ２ captures the right viewing field of the driver at a wide angle, and FoV ３ captures the left viewing field at a wide angle.

[0041] The two-dimensional image information of the three viewing fields shown in FIG. 4(a) obtained by the image sensor 1 is sent to an image arithmetic unit for image processing such as brightness adjustment and noise removal. The image after image processing is sent to an image recognition unit using machine learning, and pedestrians, automobiles, motorcycles, etc. around the vehicle are recognized. The recognition is performed using the distorted image data obtained by the far-infrared imaging device. In an in-vehicle camera for ADAS, a method of recognizing pedestrians and vehicles using an algorithm or machine learning is used. When using the recognition technology by machine learning, since the characteristics of the distortion of the image are fixed, distortion correction is not performed and the distorted image can be directly learned and recognized.

[0042] The three obtained viewing angles FoV1, FoV ２ , FoV ３Image distortion exists due to the curved mirror. Since distorted images are difficult for humans to see, image distortion correction was performed. Furthermore, the magnification, rotation, position, and brightness of the three field-of-view images were non-linearly corrected to minimize the seams between the fields of view, and then the images were stitched together horizontally to generate an image with a horizontally elongated field of view, Figure 4(b).

[0043] The image sensor area is effectively utilized by dividing the image sensor vertically and capturing different horizontal fields of view at different magnifications. As already mentioned, in-vehicle cameras require a wide horizontal field of view, high magnification in the central field of view, and a wide field of view in the peripheral field of view. The in-vehicle camera image shown in Figure 4(b) satisfies these requirements. The image shown in Figure 4(b) provides a vertical field of view of 30° and a horizontal field of view of 150°.

[0044] Here, the optical design of a wide-angle reflective / refracting system using a curved mirror will be explained in detail with reference to Figure 5. In this example, a paraboloidal mirror was used as the curved mirror, which is a mirror with a shape obtained by cutting out a part of a paraboloidal mirror obtained by rotating a parabola around the axis of symmetry. The curved mirror 3 in Figure 5 has a shape obtained by cutting out a part of the cross section of the paraboloidal mirror shown by the dashed line.

[0045] In the case of an in-vehicle camera, the point of interest requiring the highest resolution is the direction of vehicle movement. Accordingly, the axis of the rotating paraboloid is oriented in the direction of vehicle movement. Light rays (straight lines) reaching the mirror from the horizontal direction are reflected at the point of interest on the mirror: M (4) and reach the imaging lens 2. Further imaging by the imaging lens 2 forms an image on the image sensor 1. At this time, an image 6 (virtual image) of the point of interest in the field of view is formed on the outside of the imaging lens 2 of the curved mirror 3. The image 6 of the point of interest in the field of view is formed at the focal point F of the parabola.

[0046] The dashed lines show light rays reaching the curved mirror 3 from other directions within the field of view. Each light ray forms an image 5 by the mirror, and this image is projected onto the image sensor 1 by the imaging lens 2. Now consider the case where the curved mirror is a parabolic mirror of rotation. If the point of interest in the field of view is in the horizontal direction, the axis of symmetry of the curved mirror (parabolic mirror) 3 is also oriented in the same horizontal direction as the point of interest. Let the focal length of the parabolic mirror be f, the focal position be F, the center position of the imaging lens 2 be C(2-1), the point of interest on the mirror be M, and the distance between the horizontal line passing through point F and point M be h. 、 Let θ be the angle between the line FM and the axis of the paraboloid. If the curved mirror 3 is a parabolic mirror, light rays parallel to the axis of symmetry of the paraboloid are imaged at a single point without aberration. At this time, h =2f[-1 / tanθ+(1 / tan 2 θ+1) 1 / 2 ] and the distance f' between F and M is f' = f + h 2 It becomes / 4f.

[0047] The focal point F of a horizontal ray (ray from the point of interest in the field of view) is formed on the axis of the parabola. In a rotating parabolic mirror, the image of a ray coming from the axial direction is aberration-free, but as the ray is tilted off-axis, the aberration increases and the resolution deteriorates.

[0048] Here, we will explain how to set the field of view focus point. The field of view focus point is set at the location within the field of view where the highest resolution is required. As you move outward from this point, aberrations increase and resolution deteriorates. For this reason, it is generally desirable to set the field of view focus point at the center of each field of view. However, depending on the requirements of the imaging target and the optical system, there are cases where the field of view focus point and the physical center of the field of view are not aligned.

[0049] Figure 4(a) shows the points of interest within each field of view, indicated by white circles. FoV １ In this case, the point of focus in the field of view coincides with the center of the field of view, ２ FoV ３　 Therefore, the point of focus in the field of view is set at a position different from the center of the field of view. This is due to the characteristics unique to in-vehicle cameras. In in-vehicle cameras, pedestrians far away appear small in the direction the vehicle is traveling, and pedestrians closer to the side appear larger. For this reason, the central field of view (FV)１ Therefore, the center of the field of view requires the highest resolution. On the other hand, the right field of view (FV) ２ Therefore, the left side is closer to the center, and as you move to the right, the edges are visible. ２ The point of interest in the field of view is located near the left side, FoV. ３　 Then it will be set to the right side.

[0050] Furthermore, FoV ２ FoV ３ FoV １ Since the field of view is set to overlap in order to combine images, it is desirable to set the point of interest outside the overlapping area of ​​the images. ２ FoV ３ Because of image distortion, it is desirable to set the point of interest in the field of view towards the upper part of the field of view. Therefore, FoV ２ Therefore, the point of interest in the field of view is located to the upper left of the physical center of the field of view, FoV. ３ It is set to the upper right.

[0051] The imaging device 10 shown in Figure 1 achieves three fields of view by combining three curved mirrors. To achieve fields of view with different magnifications, it is necessary to combine mirrors with different focal lengths. In order to obtain an image with a single imaging lens using curved mirrors with different focal lengths, the curved mirrors must be arranged so that they are in focus on the image sensor. Next, the mirror installation method will be described.

[0052] As shown in Figure 6, consider two curved mirrors 3-1 and 3-2. For simplicity, we will again consider the case where the curved mirrors are paraboloid mirrors of revolution. In the figure, the points are denoted with subscripts 1 and 2 to distinguish the two curved mirrors.

[0053] If the points of interest 4-4 and 4-5 in the field of view are located horizontally, the axes of symmetry of the curved mirrors (parabolic mirrors) 3-1 and 3-2 are also oriented horizontally, in the same direction as the points of interest. In this case, for the light rays from the two curved mirrors to focus simultaneously on the image sensor, the focal positions F1 and F2 of the two curved mirrors must lie on the same straight line perpendicular to the optical axis of the imaging lens 2. Let δ1 and δ2 be the angles between the optical axis 2-2 of the imaging lens, the line connecting point C and point F1, and the line connecting point C and point F2. 1 cosδ1 = CF 2 cosδ 2　 The two curved mirrors need to be positioned in such a way that this occurs.

[0054] If the curved mirror 3-1 is a parabolic mirror, h 1 =2f[-1 / tanθ1+(1 / tan 2 θ1+1) 1 / 2 ] and the distance f1' between M1 and F1 is f1' = f1 + h1 2 The result is / 4f1. The same applies to the curved mirror 3-2. From the magnification required for the imaging device, the focal length of the curved mirror and f １ ,f ２ Since this is determined, in order to match the distances CF1 and CF2 from the lens center of the curved mirror to the focal point of each image, it is necessary to adjust the distance from the imaging lens center C to the point of interest M on each mirror. Furthermore, it is necessary to set the distance between CM1 so that the light rays from the object side reaching the light rays of the curved mirror closer to the lens do not interfere with the imaging lens. From the viewpoint of miniaturizing the imaging device, it is desirable to set the distance between CM1 as small as possible without interference.

[0055] Based on the above constraints, the distance between CM2 is determined by CM 2 = f １ + f ２ By setting it to -CM1, distances CF1 and CF2 coincide, and the two field images become in focus on the image sensor. The mirror arrangement can be determined similarly even when there are three or more curved mirrors.

[0056] This section describes the method for creating curved mirrors. The curved mirrors are precisely shaped by cutting acrylic resin with a diamond tip based on 3D design data. To ensure high-precision installation, a mirror block structure was adopted, where the mounting structure (including mounting components and screw holes) and the mirror surface are integrated. The mirror block is manufactured by depositing metallic aluminum vapor deposition and a protective silicon oxide film onto the reflective mirror surface.

[0057] In this embodiment, three mirrors are used. It is possible to manufacture the three mirrors separately and install them on the camera housing, but this would be costly to manufacture and would require high precision in assembling each mirror when installing them on the housing. Furthermore, there is a concern that the position of each mirror may shift due to vibration after installation, potentially degrading camera performance.

[0058] To avoid these problems, the three mirrors were formed as a single integrated mirror block 30, as shown in Figure 7. The first curved mirror 3-1, the second curved mirror 3-2, and the third curved mirror 3-3 are formed as part of a single integrated mirror block, allowing both the cutting and vapor deposition processes to be completed in a single step. When assembling to the housing, the single component makes it easier to achieve precision and reduces the likelihood of misalignment.

[0059] The mirror block shown in Figure 7 has a flat section 31 formed between the first curved mirror 3-1 and the second and third curved mirrors 3-2 and 3-3, which has the advantage of allowing for precise setting of the positional relationship with the imaging lens 2. The two holes shown on the left side of the mirror block 10 in Figure 7 are mounting screw holes 32 for attaching the mirror block to the housing of the imaging device 10.

[0060] As a second embodiment, an imaging device 10 with a different design from that of Figure 1 will be described with reference to Figure 8. As shown in Figure 8(a), the images 5-1 and 5-2 formed by the curved mirrors 3-1 and 3-2 are inclined with respect to the imaging lens 2. In such a case, even if the mirrors are simply positioned so that the images 6-1 and 6-2 formed by the mirrors at the point of interest in the field of view are equidistant from the center point of the lens, a large focus shift will occur across the entire field of view. When the image 5 formed by the mirrors has an inclined image plane in this way, a method called an inclined lens, in which the lens is tilted relative to the image sensor, becomes effective.

[0061] In the case of Figure 8(a), the intersection point of the line connecting the image sensor surface and the images 6-1 and 6-2 formed by the mirror at the point of interest in the field of view is determined, and when the angle between this intersection point and the line connecting the lens center point C and the image sensor surface is denoted as ε, the focal shift of the image 5 formed by the mirror can be minimized by tilting the imaging lens 2 by ε from the normal of the image sensor 1. Furthermore, as shown in Figure 8(b), a similar effect can be obtained by aligning the optical axes of the imaging lens 2 and the image sensor 1 and inserting an auxiliary lens 2-3, which is positioned at an angle to the optical axis, into the optical path.

[0062] Furthermore, the use of such auxiliary lenses 2-3 is particularly effective for far-infrared imaging devices. Far-infrared lenses are difficult to manufacture, and designing and manufacturing them to suit a specific purpose results in very high costs. By using commercially available general-purpose far-infrared lenses in combination with auxiliary lenses according to the application, the design flexibility can be increased. For example, when a high magnification field of view is required, it is necessary to increase not only the focal length of the curved mirror 3 but also the focal length of the imaging lens 2. With commercially available general lenses, as the focal length increases, the minimum focusing distance of the lens also increases. Therefore, it may not be possible to focus the imaging lens 2 at the focal point of the image 5 produced by the curved mirror 3. In such cases, by placing the curved auxiliary lenses 2-3 in front of the imaging lens 2, the focusing distance can be shortened and an image can be formed on the image sensor 1.

[0063] The use of such auxiliary lenses is also effective even when the auxiliary lenses are not tilted. In Figure 8(b), by setting the auxiliary lens tilt ε=0, it is possible to bring only the focal point closer. In this embodiment, a commercially available germanium lens with a focal length of 9 mm was used as the imaging lens 2, and a commercially available single lens made of ZnSe with a focal length of 25.4 mm was used as the auxiliary lens 2-3. Designing using a combination of commercially available lenses is very effective in reducing the cost of the imaging device.

[0064] As a third embodiment, we will describe an application example to in-vehicle monitoring. There are two types of systems for in-vehicle monitoring in automobiles: driver monitoring systems used to improve safety by monitoring the driver's condition, and cabin monitoring systems aimed at improving the safety and comfort of all occupants by monitoring them. By using the imaging device 10 of the present invention, multiple fields of view with different magnifications can be captured with a single camera. By capturing the driver with a high-magnification field of view and all occupants with a wide-angle field of view, a new in-vehicle monitoring system has been realized that realizes the functions of driver monitoring and cabin monitoring with a single imaging device 10.

[0065] Field of View 1 for driver monitoring and Field of View 2 for cabin monitoring ２ The optical system was designed to have the field of view shown in Figures 9(a) and (b). The imaging device 10 is installed in the front ceiling area between the driver's seat and the passenger seat. Figure 10 shows a schematic diagram of the imaging device 10. Figure 10(a) is a side view, and Figure 10(b) is a top view. In order to capture the driver at high magnification for driver monitoring, the curved mirror 3-1 is rotated to the right when viewed from above with respect to the optical axis 2-2 of the projection lens. Here, a thermal sensor was used as the image sensor 1.

[0066] Field of View (FoV) for cabin monitoring ２ This requires a wide field of view of about 120°, and the effects of aberrations become significant at the outer edges of the field of view. Therefore, a design was implemented here that uses a free-form surface mirror to suppress the degradation of resolution at the outer edges.

[0067] The design of the freeform surface mirror was based on the design of the revolutionary paraboloid mirror already described. First, the surface mirrors 3-1 and 3-4 were designed as revolutionary paraboloid mirrors to achieve the field of view shown in Figure 9. The surface mirror 3-4, which is a revolutionary paraboloid, is shown by a dashed line. In this case, the surface mirror 3-4, which has a parabolic shape with a focal length f2, has a rotation axis direction of y and an axis perpendicular to the y axis of x = x 2 This is expressed as / 4f2. At this time, h ２ =2f[-1 / tanθ2+(1 / tan 2 θ² + 1) 1 / 2 ] and at point M2 on the mirror, the radius of curvature in the tangential direction is R T = 2f2[(1+h2 / 2f2) 2 ] 3 / 2 The radius of curvature in the sagittal direction is R S = 2f2. In the case of a paraboloid of revolution, the focal position F ２ Once this is determined, the curvature on the point of interest M2 on the mirror is automatically determined.

[0068] Next, the curved mirror 3-2, which has a freeform surface shape, is designed to be tangent to the parabolic curved mirror 3-4 at point M2. In this case, the curvature on the freeform surface at M2 coincides with the curvature of the paraboloid of revolution. Conversely, the curvature of the freeform surface may be designed such that the point where tangential rays and sagittal rays passing near M2 intersect has a focal length of 2f2. In this case, the intersection point is the focal point F ２ Therefore, the curvature on M2 is equal to the curvature mentioned above. Optical simulators are used to design optical components with freeform surface shapes. Freeform surface design can involve enormous calculations, making it difficult to obtain a suitable solution. To efficiently design freeform surfaces, an approach is taken in which the initial structure is determined first, and then optimization is performed. In this process, determining an appropriate initial structure is extremely important.

[0069] With a paraboloid as the initial structure, the radius of curvature R in the tangential direction at point M2 T = 2f2[(1+h2 / 2f2) 2 ] 3 / 2 , radius of curvature R in the sagittal direction SThe most efficient approach was to determine the initial values ​​of the simulation so that = 2f2, and then optimize them using an optical simulator based on the shape of the parabolic mirror at point M2.

[0070] Figure 11 shows images taken using the imaging device 10 shown in Figure 10, which has a free-form mirror shape as the curved mirror 3-2 designed by the device. The driver's face is captured at high magnification, and it can be seen that all four occupants are captured in the wide-angle view.

[0071] Here too, a thermal sensor is used as image sensor 1. Using a thermal sensor for in-vehicle monitoring offers the following advantages: Stable monitoring is possible without being affected by ambient light (such as sunlight or the headlights of oncoming cars). Temperature measurement enables new functions in the in-vehicle monitoring system. By analyzing temperature changes around the driver's nose, it is possible to determine the driver's tension level or drowsiness, thereby improving driving safety. Furthermore, by measuring the body temperature of each occupant, it is possible to measure each occupant's sense of temperature and individually control the air conditioning according to each occupant's sense of temperature, thereby improving comfort inside the vehicle. In addition, measuring the occupants' body temperature makes it possible to manage their health and understand their health status.

[0072] From the upper image in Figure 11, it can be inferred that the body surface temperature of the other passengers is higher than that of the driver, suggesting that everyone except the driver is feeling hot. By adjusting the air conditioning temperature and airflow accordingly, the comfort level inside the vehicle can be improved.

[0073] As a fourth embodiment, we will describe an application example for drone cameras. As the uses of drones expand, there is an increasing trend to equip drones with thermal cameras in addition to visible light cameras for image capture. This is because thermal cameras can detect people and animals even in darkness by detecting body temperature. Drones are used for various purposes such as criminal investigations, searching for missing persons, surveying situations during disasters and fires, and wildlife surveys. For drone applications, cameras need to be small and lightweight. Therefore, it is expected that the range of applications for the imaging device of the present invention, which has multiple fields of view in a single camera, will expand. During investigations and searches, both a wide field of view and high magnification are often required. This is to find and confirm living organisms from a wide area.

[0074] Figure 12 illustrates an imaging system that can simultaneously capture a 360° omnidirectional field of view and a high-magnification field of view in front of the flight direction with a single camera. The imaging device 10 includes a curved mirror 3-1 for high magnification and a curved mirror 3-2 for omnidirectional imaging. The curved mirrors 3-1 and 3-2 are integrated to form a curved mirror 3. In Figure 12(a), the part of the curved mirror 3 that contributes to imaging by reflection is shown with a thick line, and in Figure 12(b), it is shown as a solid black area. The curved mirror 3-1 is a mirror with a parabolic shape and provides a high-magnification field of view FoV1 centered on the point of interest 4-4 in the diagonally downward direction. The curved mirror 3-2 provides a 360° omnidirectional field of view. In the cross-sectional direction shown in Figure 12(a), the curved mirror 3-2 has a parabolic shape with a focal length f. The curved mirror 3-2 is obtained by rotating this parabola around the optical axis 2-2 of the imaging lens. The curved mirror 3-2 is formed such that the optical axis 2-2 of the imaging lens is its axis of rotational symmetry. The curved mirror 3-2 provides a concentric 360° field of view on the image sensor 1. Here again, the two curved mirrors 3-1 and 3-2 form an integrated mirror block structure. By making the mirrors an integrated structure, the holding of the curved mirror 3-1 is made easier, and high-precision mirror arrangement is achieved.

[0075] The second point of interest on the curved mirror 3-2, M2, also lies on the circumference in the direction of the mirror's rotation, and the point of interest in the field of view on the imaging target side is in a conical direction pointing downwards in the horizontal direction. Let φ be the angle between this cone and the horizontal plane. In order to make the sagittal and tangential focal points on the second point of interest on the mirror, M2 (4-2), the mirror must be designed so that the radius R between the second point of interest on the mirror, M2, and the optical axis 2-2 of the projection lens is R = 2fcosφ. Furthermore, the mirror is positioned so that both the image 6-1 of the first point of interest in the field of view and the image 6-2 of the circular second point of interest in the field of view, lies on a plane perpendicular to the optical axis 2-2 of the imaging lens.

[0076] At this time, the image formed on the image sensor 1 by the imaging lens 2 is shown in Figure 13. The image in the direction of the drone's movement is captured at high magnification in the center (FoV1). The 360° image (FoV2) formed by the curved mirror 3-2 is a concentric circle image surrounding FoV1. The white dashed lines in FoV2 represent a low-magnification image of the same region as FoV1.

[0077] The image (video) in Figure 13 is acquired by the drone-mounted imaging device 10 and transmitted to the drone operator in real time via wireless communication. The drone operator can then control the drone while checking the situation on the video displayed on the screen.

[0078] In this scenario, the pilot can monitor the surrounding environment using FoV2 while keeping their direction of travel focused with FoV1. Furthermore, during investigations and searches, the FoV2 can detect targets, and by orienting the drone towards the target, detailed identification becomes possible.

[0079] In the embodiments described so far, a configuration has been adopted in which a virtual image formed on the outside of a convex mirror is captured by an imaging lens as the curved mirror 3. A similar imaging device can be realized even if a concave mirror is used as the curved mirror 3.

[0080] When a concave mirror is used, the real image formed inside the mirror is captured by the imaging lens. In this case as well, only the image formed by the mirror changes from a virtual image to a real image, and since the structure involves capturing the image formed by the mirror with the imaging lens, the same design principles can be applied.

[0081] This embodiment, like Embodiment 1, is an example of application to an in-vehicle far-infrared imaging system. Figure 14 shows the imaging device of this embodiment. Only a side view is shown here. The structure is almost the same, but the first and second curved mirrors 3-1 and 3-2 have been replaced from convex mirrors in Embodiment 1 (Figure 1) to concave mirrors.

[0082] In a convex mirror, the images 5-1 and 5-2 formed by the mirror are virtual images and are formed on the outside of the mirror (Figure 1). On the other hand, in the case of a concave mirror, the images 5-1 and 5-2 formed by the mirror are real images and are formed on the inside of the mirror (Figure 14). These images 5-1 and 5-2 formed by the mirror are projected onto the image sensor 1 by the imaging lens 2.

[0083] Using a concave mirror makes it possible to magnify the image. (Figure 14 shows the field of view (FV)). １ The field of view (FoV) in Figure 1 １ Compared to the previous method, it can be seen that the angle is narrower and the magnification is higher. This is an effective method because it allows the magnification of the imaging device to be increased without increasing the magnification of imaging lens 2.

[0084] However, since the real image formed using a concave mirror is located inside the curved mirror, the distance between the mirror and the imaging lens increases, resulting in a larger imaging device 10. Therefore, the shape of the mirror must be appropriately selected depending on the application.

[0085] Next, another embodiment using a concave mirror will be explained with reference to Figure 15. This is an example of using a concave mirror in a drone camera, as shown in Example 4. In this configuration, the first curved mirror 3-1 for high-magnification imaging is a concave mirror, and the second curved mirror 3-2 for 360° image acquisition is a convex mirror.

[0086] The image 6-1 of the first point of interest in the field of view, formed by the first curved mirror 3-1, is formed on the inside of the first curved mirror 3-1. On the other hand, the image 6-2 of the second point of interest in the field of view, formed by the second curved mirror 3-2, is formed on the outside of the second curved mirror 3-2.

[0087] At this time, by appropriately arranging the first curved mirror 3-1 and the second curved mirror 3-2, it becomes possible to make the image 6-1 of the first point of interest in the field of view and the image 6-2 of the second point of interest in the field of view appear to be on the same image plane as viewed from the image sensor 1.

[0088] As described above, by appropriately arranging multiple mirrors according to their shapes, it becomes possible to make the images formed by multiple mirrors exist on the same image plane. This means that the focal points of the fields of view covered by each mirror coincide. By controlling the orientation, size, and shape of the mirrors, it becomes possible to set the magnification and direction of each field of view with a high degree of freedom.

[0089] In the embodiments described so far, a thermal sensor was used as image sensor 1, but it is also possible to use a visible image sensor or a near-infrared image sensor. The reason for using a thermal sensor here is that, because thermal sensors are more expensive than visible image sensors, the advantage of obtaining multiple fields of view with a single thermal sensor can be maximized.

[0090] Imaging devices with multiple fields of view at different magnifications can satisfy two seemingly contradictory requirements: wide angle and high magnification. Since functions previously achieved using multiple imaging devices can now be realized with a single device, cost reduction is particularly significant for imaging devices in the far-infrared region, where image sensors are expensive.

[0091] Configuration diagram of the imaging device of Example 1. (a) Side view, (b) Top view. Panoramic image of an intersection taken from the driver's seat. Diagram showing the desirable field of view of an in-vehicle camera. (a) When turning left, (b) When turning right. Image taken with the imaging device of Example 1. (a) Image on the image sensor, (b) Image integrating the three fields of view. Diagram explaining a reflective / refracting wide-angle system using a curved mirror. Diagram explaining a reflective / refracting wide-angle system using two curved mirrors. Mirror block. Configuration diagram of the imaging device of Example 2. Diagram showing the field of view of the imaging device of Example 3. (a) Side view, (b) Enlarged view of the imaging lens section. Configuration diagram of the imaging device of Example 3. (a) Side view, (b) Top view. Image taken with the imaging device of Example 3. Configuration diagram of the imaging device of Example 4. (a) Side view, (b) Top view. Image taken with the imaging device of Example 4. Configuration side view of the imaging device of Example 5. Configuration side view of the imaging device of Example 6.

[0092] 1 ... Image sensor 2 ... Imaging lens 2-1 ... Center point of the imaging lens: C 2-2 ... Optical axis of the imaging lens 2-3 ... Auxiliary lens 3 ... Curved mirror 3-1 ... First curved mirror 3-2 ... Second curved mirror 3-3 ... Third curved mirror 3-4 ... Parabolic curved mirror 4 ... Point of interest on the mirror: M 4-1 ... Point of interest on the first mirror: M1 4-2 ... Point of interest on the second mirror: M2 4-3 ... Point of interest on the third mirror: M3 4-4 ... Point of interest in the first field of view 4-5 ... Point of interest in the second field of view 4-6 ... Point of interest in the third field of view 4-7 ... Point of interest in the field of view 5 ... Image formed by the mirror 5-1 ... Image formed by the mirror of the first field of view 5-2 ... Image formed by the mirror of the second field of view 5-3 ... Image formed by the mirror of the second field of view 6 ...Image of the point of interest in the field of view as seen by the mirror 6-1 ...Image of the first point of interest in the field of view as seen by the mirror 6-2 ...Image of the second point of interest in the field of view as seen by the mirror 6-3 ...Image of the second point of interest in the field of view as seen by the mirror 10 ...Imaging device 21 ...Automobile 22 ...Bicycle 23 ...Pedestrian 24 ...Field of view of a conventional in-vehicle thermal camera 25 ...Desired field of view 26 ...Particularly desirable field of view 30 ...Mirror block 31 ...Flat surface 32 ...Mounting screw holes

Claims

1. An imaging device that uses a single image sensor to simultaneously capture multiple different fields of view divided into a field of view at different magnifications, comprising, in order from the image side to the field of view side, one image sensor, one imaging lens, and a plurality of curved mirrors positioned off-axis with respect to the optical axis of the imaging lens so as to pass through the imaging lens and form images on different regions on the image sensor, wherein at least one of the plurality of curved mirrors has a different shape from the other curved mirrors.

2. The imaging device according to claim 1, wherein, when the point requiring the highest resolution within each of the multiple different fields of view of the divided field of view is defined as the field of view point of interest, the point on the curved mirror corresponding to the field of view point of interest is defined as the mirror point of interest M, the lens center point C of the imaging lens is defined as the focal point F of the image of the field of view point of interest formed by the mirror, which is formed by light rays reaching from the field of view point of interest to the vicinity of the mirror point of interest M, the angle between the optical axis of the imaging lens and the straight line MC is defined as δ, and the distance between point F and point C is defined as FC, a plurality of curved mirrors are arranged such that the distance FCcosδ is substantially the same for the plurality of curved mirrors.

3. The imaging apparatus according to claim 1 or 2, wherein at least two of the plurality of curved mirrors have an integrated mirror block structure.

4. The imaging device according to claim 2, wherein at least one of the plurality of curved mirrors is a parabolic mirror of rotation, and the axis of rotational symmetry of the parabolic mirror of rotation is positioned in substantially the same direction as the direction from the point of interest M on the mirror toward the point of interest in the field of view.

5. The imaging apparatus according to claim 1 or 2, wherein the optical axis of the imaging lens is arranged inclined with respect to the normal of the image sensor.

6. At least one of the plurality of curved surface mirrors is a free-form surface mirror, and the radius of curvature in the tangential direction of the point of interest M on the mirror of the free-form surface mirror is R T , the radius of curvature in the sagittal direction is R S Let f be the focal length of the paraboloid of revolution that is tangent to the free surface at the point of interest M on the mirror, and let h be the distance between the axis of symmetry of the paraboloid of revolution and the point of interest M on the mirror. T =2f[(1+h / 2f) 2 ] 3 / 2 R S The imaging apparatus according to claim 1 or 2, wherein = 2f.

7. The imaging device according to claim 1 or 2, wherein at least one of the plurality of curved mirrors is a curved mirror obtained by rotating a parabola with focal length f around a line that makes an angle φ with the axis of symmetry of the parabola, and the radius R of the circle formed on the curved mirror at a point of interest M on the mirror is R = 2fcosφ.

Images