Surround view imaging system with simultaneous TOF and RGB image acquisition
Separating the TOF and RGB spectra by a cylindrical reflective refractive lens system, solving the signal interference and occlusion problems in the prior art, realizing high-resolution image acquisition on 360° HFOV, and improving the stability and resolution of the imaging system.
Patent Information
- Application Number
- CN202380078106.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-15
- Publication Date
- 2025-07-11
AI Technical Summary
Existing surrounding view imaging systems are difficult to achieve full detector resolution of TOF and RGB data simultaneously on a complete 360° horizontal field of view, and the prior art has signal interference and image occlusion problems.
The cylindrical reflective refractive lens system is used to separate imaging light from different spectral ranges through the top surface spectral filter element, and project onto the TOF and RGB detectors respectively. The monolithic reflective refractive lens design avoids signal interference and achieves 360° HFOV.
It realizes high-resolution TOF and RGB image acquisition without occlusion on 360° HFOV, reducing the preprocessing requirements for image processing and improving the stability and resolution of the imaging system.
Smart Images
Figure CN120303600A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a surround view imaging system for three-dimensional (3D) imaging of the surrounding environment of a system, and particularly to an imager for such a surround view imaging system, which enables simultaneous time-of-flight (TOF) and optical (RGB) image acquisition with full sensor resolution. Background Art
[0002] For 3D imaging systems or sensors capable of locating objects in the 3D surrounding environment of a system, there are different methods available based on various technologies, such as light detection and ranging (LiDAR), time-of-flight (ToF, direct and indirect versions), amplitude or frequency modulation illumination, structured light, etc. Such systems are typically present in autonomous mobile robots (AMRs), industrial mobile robots (IMRs), and automated guided vehicles (AGVs) such as forklift trucks, forklifts, cars, drones, for collision avoidance, obstacle detection, passenger monitoring, and observing restricted areas of machines and robots. Surround view imaging systems can also be used in collaborative robots, security, and surveillance camera applications.
[0003] If the system is optical and uses an array detector (e.g., CMOS sensor, CCD sensor, photodiode array) to avoid movement of components in the system, then the receiving lens that images the surrounding environment onto the associated image detector is a highly critical element. The lens must allow for high resolution over a wide field of view in the horizontal and vertical directions. At the same time, it should have uniform imaging characteristics and high light flux without vignetting to achieve a large detection range.
[0004] For a wide horizontal field of view (HFOV), e.g., greater than 120 degrees, a fish-eye lens can be used in an upright position. However, conventional fish-eye lenses have several drawbacks, such as high incident angles and related coating problems. Another problem is that the very wide field of view combines low resolution, low f-number, and vignetting caused by off-axis illumination. These drawbacks can be avoided by using a catadioptric lens system in which a mirror and a lens are combined to form an image.
[0005] By using a wide-angle lens (such as a fish-eye lens or a rectilinear lens) as the first lens of the lens system in the corresponding imaging system, a surround-view image can be generated. The wide-angle lens can have a field of view (AOV) of more than 180°, for example, in the maximum zenith angle range in the vertical direction where the lens can provide an image. A lens with an AOV of more than 180° is called an ultra-wide-angle lens. A field of view of up to about 300° can be achieved. In a typical axially symmetric imaging system, the imaged azimuth range, for example, in the horizontal direction, is usually 360°, thus allowing a surround view to be achieved in the azimuth direction. Therefore, using an ultra-wide-angle lens can image a solid angle Ω of up to about 3π steradians. Wide-angle lenses typically exhibit strong curvilinear barrel distortion, while rectilinear lenses can be optically corrected to some extent. Optical barrel distortion correction can also be included in the design of the relevant lens system. Therefore, a lens system with an AOV greater than 180° is called an ultra-wide-angle lens system.
[0006] To further improve the accuracy and reliability of 3D imaging, multiple techniques can be combined in a single surround-view imaging system. In particular, the combination of advanced TOF technology and RGB image acquisition has proven to be particularly beneficial. In such a combined imaging system, the images captured by different techniques must have significant overlap to allow for fast and efficient image processing for subsequent analysis of the combined image data.
[0007] Therefore, such a surround-view imaging system typically employs a single image detector that is capable of detecting two signals of the imaging light received by a common beam path. The acquired images then enter a common FOV. In such a dual-mode detector, each pixel typically has a set of four independent sub-pixels, namely sub-pixels sensitive to red, green, blue, and infrared light. However, such a detector can only provide reduced TOF and RGB data resolution because the TOF and RGB sub-pixels of the pixel are positioned side by side, which limits the resolution of the optical system (see Figure 1 ).
[0008] An alternative method that can use the full resolution of the detector is the so-called side-by-side arrangement of each detection system. However, these detection systems must be arranged such that they also belong to a common FOV, which means they must be aligned in sequence in a common plane, such as a horizontal plane. However, this way of side-by-side arranging dedicated TOF and RGB detectors cannot achieve a complete 360° horizontal FOV because the FOV of the rear detector will be partially blocked by the front detector. In addition, due to the slightly different perspectives of the scenes in the surrounding environment of the system (see Figure 2 ), the image information still needs to be aligned and preprocessed in software later.
[0009] The objective problem of the present invention relates to the problem of simultaneously acquiring TOF and RGB data at full detector resolution over a complete 360° HFOV. Accordingly, a surround view imaging system should be provided that avoids or at least reduces the problems of the prior art combined TOF / RGB imaging systems. Summary of the Invention
[0010] The present invention solves this objective problem by providing an imager for a surround view imaging system as defined in claim 1. A surround view imaging system comprising an imager according to the present invention is further provided.
[0011] The imager for a surround view imaging system according to the present invention comprises a first image detector and a cylindrical refractive-reflective lens system that forms an internal volume having an entrance aperture, a top surface, and a bottom surface. In the field of view of the imager, imaging light from the surroundings of the imager enters the internal volume through the entrance aperture, is first reflected towards the top surface by the bottom surface, secondly reflected back to the bottom surface by the top surface, and exits the internal volume through a bottom aperture in the bottom surface towards the first image detector for detecting the imaging light. The spectrum of the imaging light comprises a first spectral range and a second spectral range different from the first spectral range. The top surface comprises a spectral filtering element that reflects only the imaging light of the first spectral range back to the bottom surface, but allows the imaging light of the second spectral range to transmit.
[0012] Preferably, the imager further comprises a second detector for detecting the transmitted imaging light. Preferably, the imager further comprises a first optical system for projecting an image of the surroundings in the first spectral range onto the first image detector. Preferably, the imager further comprises a second optical system for projecting an image of the surroundings in the second spectral range onto the second image detector. In a particularly preferred embodiment, the first image detector is a detector for the visible spectral range (RGB detector) and the second detector is a TOF detector, or the first image detector is a TOF detector and the second detector is a detector for the visible spectral range (RGB detector).
[0013] An imager should be understood as a device capable of receiving, focusing, and detecting imaging light entering the imager from its surrounding environment. The imager thus typically includes at least one (preferably a 360-degree annular circumference) entrance aperture adjacent to the surrounding environment, a lens or other optical element for generating an image of the surrounding environment, and a related image detector for detecting the generated image of the surrounding environment for further processing. Since the generation of the image is the most important aspect in ensuring good image quality, instead of using a single lens or optical element, a lens system (or generally a system of optical components) for correcting the occurring aberrations can be used in the imager. The imager can be a device that uses ambient light (e.g., 3D visible light or infrared light) for imaging or can be particularly adapted to image reflected light (illumination light) from an illumination source or illuminator as imaging light (e.g., flash LIDAR).
[0014] In a combined ToF / RGB imaging system, the spectrum of the imaging light typically includes a first and a second spectral range. In particular, the first spectral range can refer to the visible light (VIS) spectrum (or at least a part of said spectral range), and the second spectral range can refer to the infrared light (IR / SWIR) spectrum (or at least a part of said spectral range), and vice versa. In particular, the first spectral range can belong to RGB imaging and the second spectral range can belong to TOF imaging. These two spectral ranges must be separated to avoid any signal interference between the two systems.
[0015] The top surface of the refractive-reflective lens system includes a spectral filtering element that reflects imaging light only from the first spectral range back to the bottom surface but allows imaging light of the second spectral range to transmit. Thus, a beam splitter is included in the beam path, which allows environmental images with respective spectra to be generated on different detectors. Thus, the two images can belong to a common FOV, and the two detectors do not occlude each other.
[0016] The main idea of the present invention is to use a custom lens to collect RGB and TOF data simultaneously. The reflected light from an active or passive illumination scenario can be collected over a 360° HFOV. The light can be reflected on a first mirror towards a second optical element. This optical element can be a transmitter for the VIS spectrum (RGB) and a reflector for the NIR / SWIR spectrum (TOF). For example, a dedicated RGB detector can collect VIS photons above the first mirror. A dedicated TOF detector can collect NIR / SWIR photons in the area below the first mirror. The detectors can be optimized individually by considering resolution, sensor size, pixel size and interface, as well as performance. The FOV and other parameters of the TOF and RGB detectors can be optimally adapted to the corresponding requirements of a specific application. The two detectors share the beam path and belong to the same 360° HFOV, can observe without any blind spots, and can capture the complete scene around the system over a 360° HFoV and a fully adjustable vertical FOV (VFoV). Since the information is already matched due to the same FOV, no preprocessing of the data is required.
[0017] Preferably, the cylindrical catadioptric lens system is formed by a cylindrical monolithic catadioptric lens having a body filling the internal volume and having a sleeve including an entrance aperture, a top surface, and a bottom surface. Optionally, the cylindrical catadioptric lens system can be based on an arrangement of individual optical components such as mirrors, beam splitters, and deflectors. In this case, the internal space can remain free.
[0018] For a preferred embodiment of the cylindrical monolithic catadioptric lens, the imaging light is first reflected by a circumferential first aspherical lens region arranged around the center of the bottom surface, secondly by a second aspherical lens region arranged at the center of the top surface, and exits the internal volume towards the image detector through a third aspherical lens region located at the center of the bottom surface.
[0019] The imaging light can thus enter the monolithic catadioptric lens via the cylindrical side of the monolithic catadioptric lens and can then be reflected successively by two aspherical mirrors (or mirror surfaces). The first aspherical mirror surface interacting with the light can be a Forbes aspherical (G.W. Forbes, “Shape specification for axially symmetric optical surfaces”, Opt. Express 15(8), 5218 - 5226 (2007)), while the other mirror surface can exhibit a standard aspherical description. As a result of using the Forbes aspherical, improved optical performance can be obtained for the above - mentioned surfaces.
[0020] Imaging light can exit the internal space of the monolithic catadioptric lens via a third aspherical surface (e.g., a standard aspherical surface), which adds an additional degree of freedom for the ability to correct optical aberrations. Compared to a typical fish-eye lens, additional benefits of the monolithic design are a moderate surface tangent slope and angle of incidence, as well as a smaller element diameter. Compared to a solution with a single mirror element, the monolithic lens design provides simple system assembly and can be manufactured more precisely and with lower tolerances.
[0021] In contrast to a standard fish-eye lens, the catadioptric lens design limits the field of view of the imager in the vertical direction to avoid saturation and overexposure of the associated image detector. In particular, the system can have a horizontal and vertical field of view of 360 degrees × 60 degrees. In the case where the catadioptric lens is in the vertical position, for example, the 60 degrees can be divided into 45 degrees upward and 15 degrees downward from the horizontal plane. However, even wider horizontal and vertical fields of view of up to 360 degrees × 120 degrees can also be achieved with such a lens. By limiting the field of view to the desired angular range, only the imaging light from the relevant area of the surrounding environment can enter the lens and the imager, respectively. The smaller vertical field of view thus reduces the likelihood of detector saturation due to accidentally captured ambient and scattered light. In particular, for ambient light reflected at a flat angle of incidence (e.g., the bright reflection of the evening sun on a wet road), the entry of ambient light into the imager can be avoided.
[0022] For example, the monolithic catadioptric lens can be designed to have an f-value of 1.5 across the entire field of view without vignetting. The preferred f-value ranges between 1.2 and 1.8, more preferably between 1.4 and 1.6. Due to the compact monolithic design of the lens, aberrations can already be effectively corrected during the production of the lens, and no complex and error-prone post-assembly process is required. This also ensures good long-term stability of the imager and makes the lens relatively independent of changes in external environmental parameters such as temperature or humidity.
[0023] Since the monolithic already includes three aspherical surfaces, the rest of the optical system can be implemented with only simple spherical lenses while still ensuring good optical performance (e.g., MTF, distortion, etc.) at a moderate cost. The distortion can be selected such that the vertical and horizontal resolutions (at least approximately) are the same at an image detector with quadratic pixels. Additionally, the distortion can be specifically generated to obtain the required resolution in a specific ROI. However, to further improve the optical characteristics of the lens, the area of the sleeve can also include additional aspherical shapes, and the imaging light from the surrounding environment of the imager enters the body in this area of the sleeve. In this case, there can be four aspherical surfaces on the lens to obtain higher performance and / or lower or improved distortion characteristics.
[0024] Preferably, the entrance aperture of the imager includes an anti-reflection coating configured to transmit the full spectrum of the imaging light. Preferably, the spectral filtering element includes a dielectric layer or a grating.
[0025] Preferably, the first detector and the second detector are arranged opposite to each other, and their effective surfaces are aligned parallel to the vertical axis of the imager.
[0026] In a preferred embodiment, the image detector may have an active detection area adapted to the image size or a specifically defined region of interest (ROI). Since the central region of the image may be irrelevant to imaging and this central region may correspond to the zenith angle outside the effective FOV of the imager, these regions of the image detector can be completely omitted or ignored either from the image readout or by selective mapping to the effective active detector surface. This has the advantage that the passive regions of the image detector do not saturate due to accidentally captured ambient light and scattered light. Additionally, since it is not necessary to read out the unimportant detector regions, for a specific detector configuration, the effective frame rate of a specific type of detector can be increased. With a higher frame rate, the accumulation of photo-induced charge carriers in the individual pixels of the detector can be reduced, such that the signal-to-noise ratio (SNR) of the detector can be optimized for image detection over a wide dynamic range without using high dynamic range (HDR) techniques.
[0027] Further preferred embodiments of the present invention result from the features mentioned in the dependent claims.
[0028] Unless otherwise specified in particular cases, the various embodiments and aspects of the present invention mentioned in this application can be combined with each other to obtain advantages. Description of the Drawings
[0029] Hereinafter, the present invention will be described in further detail with reference to the drawings. The examples given are suitable for describing the present invention. The drawings show:
[0030] Figure 1 is a schematic diagram of a pixel array in a prior art dual-mode detector for simultaneous TOF and RGB imaging;
[0031] Figure 2 is a schematic diagram of a prior art side-by-side configuration of two independent detectors for simultaneous TOF and RGB imaging;
[0032] Figure 3 is a schematic diagram of an exemplary embodiment of the imager of the present invention; and
[0033] Figure 4 is a schematic diagram of optimizing the image resolution and FOV of the imager of a surround view imaging system using different alignments of the image on the detector. Detailed Description of the Invention
[0034] Figure 1 Shows a schematic diagram of a pixel array in a prior art dual - mode detector for simultaneous TOF and RGB imaging. The combination of TOF and RGB pixels at the sensor level reduces resolution and degrades image quality. A sensor of 640×480 pixels (e.g., Panasonic GC1N) provides only a TOF resolution of 320×240, which is far from sufficient for 360° HFOV imaging that requires significantly higher resolution.
[0035] Figure 2 Shows a schematic diagram of a prior art side - by - side configuration of two independent detectors for simultaneous TOF and RGB imaging. However, this way of arranging dedicated TOF and RGB detectors side - by - side cannot achieve a full 360° horizontal FOV because the FOV of the rear detector is partially blocked by the front detector. In addition, due to the slightly different perspectives of the scenes in the surrounding environment of the system, the image information still needs to be aligned and pre - processed later in software.
[0036] Figure 3 Shows a schematic diagram of an exemplary embodiment of an imager 10 of the present invention. The imager 10 includes a first image detector 12 and a cylindrical catadioptric lens system that forms an internal volume 30 having an entrance aperture 32, a top surface 34, and a bottom surface 36. In the field of view FOV10 of the imager, imaging light B from the surrounding environment of the imager 10 enters the internal volume 30 through the entrance aperture 32, is first reflected by the bottom surface 36 towards the top surface 34, is secondly reflected back by the top surface 34 to the bottom surface 36, and exits the internal volume 30 through a bottom aperture in the bottom surface 36 towards the first image detector 12 for detecting the imaging light B. The spectrum of the imaging light B includes a first spectral range and a second spectral range different from the first spectral range. The top surface 34 includes a spectral filtering element 28 that reflects only the imaging light B in the first spectral range back to the bottom surface 36 but allows the imaging light B in the second spectral range to transmit.
[0037] The cylindrical catadioptric lens system can preferably be formed by a cylindrical monolithic catadioptric lens 20 having a body that fills the internal volume 30 and a sleeve that includes an entrance aperture 32, a top surface 34, and a bottom surface 36. In this case, the imaging light B can first be reflected by a circumferential first aspherical lens region 22 arranged around the center C1 of the bottom surface 36, can secondly be reflected by a second aspherical lens region 24 arranged at the center C2 of the top surface 34, and exits the internal volume 30 through a third aspherical lens region 26 located at the center C1 of the bottom surface 36 towards the image detector 12.
[0038] The corresponding catadioptric lens 20 can thus comprise four optically effective surfaces on which the imaging light B is redirected while propagating through the monolithic body 30. The sleeve 32 and the third aspherical lens region 26 should be highly transparent to the imaging light B. On these surfaces, when the refractive index of the catadioptric lens 20 differs from the refractive index of the surrounding environment, the imaging light is redirected by diffraction. Preferably, the catadioptric lens 20 is made of a plastic material with a high refractive index (transparent in the relevant spectral range of the imaging light B), such as acrylic, polystyrene, polycarbonate, cycloolefin polymer (COP), or a composite material made of these materials. However, any material that is transparent in the relevant spectral range of the imaging light B can be used.
[0039] In the first aspherical lens region 22 and the second aspherical lens region 24, the imaging light B can be redirected by reflection. This means that in these regions, the respective surfaces of the body 30 can act as mirrors for the incident imaging light B. Preferably, the mirrors can be fabricated by coating the corresponding surfaces with a metal or dielectric layer (mirror surface). The dielectric layer can be a dielectric stack designed to provide high reflectivity in the relevant spectral range of the imaging light B. The use of reflective surfaces generally prevents the occurrence of scattered light in the catadioptric lens 20, which might accidentally enter subsequent sections of the imager 10. In other words, the catadioptric lens 20 is a stable and compact optical component that is inexpensive and easy to produce, and reduces the risk of saturation and overexposure of the associated image detector 12 by avoiding the occurrence of stray light inside the catadioptric lens 20.
[0040] The illustrated imager 10 further includes a first optical system 14 for projecting an image of the environment in a first spectral range onto a first image detector 12. The imager further includes a second detector 42 for detecting the transmitted imaging light B and a second optical system 44 for projecting an image of the environment in a second spectral range onto the second image detector 12. The first detector 12 and the second detector 42 are arranged opposite each other, and their effective surfaces are aligned parallel to the vertical axis of the imager 10.
[0041] The first optical system is exemplarily shown as a lens stack located between the catadioptric lens 20 and the image detector 12. In particular, the shown lens stack includes eight spherical lenses for further image projection. Since the shown monolithic lens 20 includes three aspheres, the remaining part of the optical system can be implemented with only standard spherical lenses while still ensuring good optical performance (e.g., MTF, distortion, etc.) at a moderate cost. The imager 10 may further include an additional band-pass filter, which may preferably be arranged between the optical systems 14, 44 and the respective image detectors 12, 42. The additional band-pass filter may cut off spectral components of the illumination light that are not relevant for image generation or that may cause saturation and overexposure of the image detector 12. The second optical system 44 may be identical or at least comparable to the first optical system 14. Since the two optical paths are very similar, the optical requirements for imaging may also be similar. However, adjustments may be required due to the different spectral ranges involved. The first image detector 12 may be a detector for the visible spectral range (RGB), and the second detector 42 may be a time-of-flight (ToF) detector, and vice versa. Preferably, the entrance aperture 32 includes an anti-reflection coating configured to transmit the full spectrum of the imaging light B. In another preferred embodiment, the spectral filtering element 28 includes a dielectric layer or a grating.
[0042] Figure 4 A schematic diagram shows the use of different alignments of the image on the detector to optimize the image resolution and FOV of the imager of a surround view imaging system. The projection of the scene onto the detector can be adjusted to obtain a smaller but higher-resolution horizontal / vertical FOV. While in example a), the FOV of the imager is fully imaged on the effective surface of the detector, example b) shows an alignment where the horizontal FOV is slightly reduced to θ = 270° at a detector with an effective surface of the same size. Example c) shows another example with a smaller detector and a different aspect ratio. On this particular detector, the horizontal FOV is even further reduced to θ = 180°. However, the full vertical FOV can still be covered by the detector.
[0043] List of reference numerals
[0044] 10 Imager
[0045] 12 First image detector
[0046] 14 First optical system
[0047] 20 Catadioptric lens
[0048] 22 First aspherical lens region
[0049] 24 Second aspherical lens region
[0050] 26 Third aspherical lens region
[0051] 28 Optical filter element
[0052] 30 Internal volume (body)
[0053] 32 Entrance aperture (sleeve)
[0054] 34 Top surface
[0055] 36 Bottom surface
[0056] 42 Second image detector
[0057] 44 Second optical system
[0058] B Imaging light
[0059] C1 Center (bottom surface 36)
[0060] C2 Center (top surface 34)
[0061] FOV10 Field of view of the imager
[0062] RGB Red, Green, Blue
Claims
1. An imager (10) for a surround view imaging system, comprising a first image detector (12) and a cylindrical refractive-reflective lens system that forms an internal volume (30) having an entrance aperture (32), a top surface (34), and a bottom surface (36); wherein in the field of view (FOV10) of the imager, imaging light (B) from the surroundings of the imager (10) enters the internal volume (30) through the entrance aperture (32), is first reflected by the bottom surface (36) towards the top surface (34), then reflected back by the top surface (34) to the bottom surface (36), and exits the internal volume (30) through a bottom aperture in the bottom surface (36) towards the first image detector (12) for detecting the imaging light (B). Characterized in that, the spectrum of the imaging light (B) includes a first spectral range and a second spectral range different from the first spectral range, wherein the top surface (34) includes a spectral filtering element (38) that reflects only the imaging light (B) in the first spectral range back to the bottom surface (36), but allows the imaging light (B) in the second spectral range to transmit through.
2. The imager (10) according to claim 1, wherein, The cylindrical refractive-reflective lens system is formed by a cylindrical monolithic refractive-reflective lens (20) having a body that fills the internal volume (30) and a sleeve including the entrance aperture (32), the top surface (34), and the bottom surface (36).
3. The imager (10) according to claim 1 or 2, wherein, The imaging light (B) is first reflected by a circumferential first aspherical lens region (22) arranged around the center (C1) of the bottom surface (36), then reflected by a second aspherical lens region (24) arranged at the center (C2) of the top surface (34), and exits the internal volume (30) through a third aspherical lens region (26) located at the center (C1) of the bottom surface (36) towards the image detector (12).
4. The imager (10) according to any one of the preceding claims, wherein, The imager (10) further includes a first optical system (14) for projecting an image of the surroundings in the first spectral range onto the first image detector (12).
5. The imager (10) according to any one of the preceding claims, further comprising a second detector (42) for detecting the transmitted imaging light (B).
6. The imager (10) according to claim 5, wherein, The imager (10) further includes a second optical system (44) for projecting an image of the surroundings in the second spectral range onto the second image detector (42).
7. The imager (10) according to claim 5 or 6, wherein, The first image detector (12) is a detector for the visible spectral range and the second detector (42) is a time-of-flight detector, or wherein, the first image detector (12) is a time-of-flight detector and the second detector (42) is a detector for the visible spectral range.
8. The imager (10) according to any one of the preceding claims, wherein, The entrance aperture (32) includes an anti-reflection coating configured to transmit the full spectrum of the imaging light (B).
9. The imager (10) according to any one of claims 5 to 8, wherein, The first detector (12) and the second detector (42) are arranged opposite to each other, and their effective surfaces are aligned parallel to the vertical axis of the imager (10).
10. The imager (10) according to any one of the preceding claims, wherein, The spectral filtering element (28) includes a dielectric layer or a grating.