Opto-mechanics of panoramic capture devices with adjacent cameras

Abstract

To improve a panoramic multi-camera device and its low-parallax camera with respect to optical and optomechanical design, as well as other aspects.SOLUTION: A low-parallax imaging device includes a plurality of imaging lens elements arranged to capture adjacent fields of view. In some examples, adjacent imaging lens elements may contact at datum features to maintain a desired spacing. The spacing may allow for partial overlapping of low-parallax volumes associated with the respective imaging lens elements.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 865,741, entitled "Opto-Mechanics of Panoramic Capture Devices with Abutting Cameras," filed June 24, 2019; U.S. Provisional Patent Application No. 62 / 952,973, entitled "Opto-Mechanics of Panoramic Capture Devices with Abutting Cameras," filed December 23, 2019; U.S. Provisional Patent Application No. 62 / 952,983, entitled "Multi-camera Panoramic Image Capture Devices with a Faceted Dome," filed December 23, 2019; and U.S. Provisional Patent Application No. 62 / 972,532, entitled "Integrated Depth Sensing and Panoramic Camera System," filed February 10, 2020, each of which is incorporated by reference in its entirety.

[0002] This disclosure relates to a panoramic low-parallax multi-camera capture device having multiple adjacent, abutting polygonal cameras. This disclosure also relates to an optomechanical design for the cameras that captures incident light from polygonal fields of view to form a polygonal image. [Background technology]

[0003] Panoramic cameras have substantial value due to their ability to simultaneously capture wide-angle images. The earliest such example is the fisheye lens, an ultra-wide-angle lens that produces strong visual distortion while capturing a wide panoramic or hemispherical image. While the field of view (FOV) of a fisheye lens is typically between 100 and 180 degrees, this approach has been extended to even wider angles, including a range of 220 to 270 degrees, such as that proposed by Y. Shimizu in U.S. Patent 3,524,697. Alternatively, there are mirror- or reflection-based cameras that capture circular panoramic images, such as the system proposed by P. Greguss in U.S. Patent 4,930,864. While these technologies continue to evolve, they struggle to provide full hemispherical or spherical images at the resolution and image quality currently required by modern applications.

[0004] As another alternative, panoramic multi-camera devices, in which multiple cameras are arranged around or around the circumference of a sphere, are becoming increasingly popular. However, in most of these systems, including those described in U.S. Patents 9,451,162 and 9,911,454 to A. Van Hoff et al., both of Jaunt Inc., the cameras are sparsely distributed on the exterior surface of the device. To capture a complete 360-degree panoramic image, including the gaps or seams between adjacent individual cameras, the cameras then have extended FOVs that overlap with each other. In some cases, as much as 50% of the camera FOV or resolution may be used for overlap between cameras, which also creates substantial parallax differences between the captured images. Parallax is the visual perception that the position or orientation of an object appears different when viewed from different directions. Then, in subsequent image processing, both the excess image overlap and parallax differences complicate and significantly slow efforts to properly combine, tile, or stitch together acceptable images from images captured by adjacent cameras.

[0005] There are also panoramic multi-camera devices in which multiple cameras are arranged around or around the circumference of a sphere, with adjacent cameras touching along part or all of their adjacent edges. As an example, U.S. Patent 7,515,177 to K. Yoshikawa describes an imaging device with multiple adjacent image pickup units (cameras). Images are collected from cameras with overlapping fields of view to compensate for mechanical errors.

[0006] More broadly, in multi-camera devices, mechanical variations in the assembly and alignment of individual cameras and cameras adjacent to one another can cause actual physical variations in both the cameras themselves, seam width, and the parallelism of camera edges along the seam. These variations can then affect the FOV captured by each camera, the parallax error in images captured by adjacent cameras, the extent of "blind spots" in the FOV corresponding to the seam, the seam width, and the amount of image overlap that needs to be corrected. Thus, there are opportunities to improve the optical and optomechanical design of panoramic multi-camera devices and their low-parallax cameras, as well as other aspects. [Brief explanation of the drawings]

[0007] [Figure 1] 1 shows a portion of a multi-camera device capture device, and specifically the 3D views of its two adjacent cameras. [Figure 2A] 1 shows a cross section of a portion of a camera lens assembly, including lens elements and ray paths. [Figure 2B] 1 shows a cross section of a portion of a camera lens assembly, including lens elements and ray paths. [Figure 3] 1 shows a cross-sectional view of a portion of a typical multi-camera capture device, illustrating FOV overlap, field of view, overlap, seams, and blind areas. [Figure 4] 1 shows two polyhedral shapes, a regular dodecahedron and a truncated icosahedron, for which multi-camera capture devices can be designed and manufactured. [Figure 5A]Figure 5B shows the optical geometry for the field of view for adjacent hexagonal and pentagonal lenses, as might occur in a device with a truncated icosahedral geometry. Figure 5B shows the enlarged region of Figure 5A in greater detail. [Figure 5B] Figure 5B shows the optical geometry for the field of view for adjacent hexagonal and pentagonal lenses, as might occur in a device with a truncated icosahedral geometry. Figure 5B shows the enlarged region of Figure 5A in greater detail. [Figure 5C] FIG. 5C shows an example of a paraxial NP point or low parallax (LP) quantity located near both the entrance pupil and the device center. [Figure 5D] The parallax difference for two adjacent cameras is shown relative to the perspective center. [Figure 5E] 1 shows the front color at the edge of the outer compressor lens element. [Figure 6] 1 shows a distortion correction curve plotted on a graph showing the percentage of distortion versus fractional field. [Figure 7] The fields of view for adjacent cameras are shown, including both core and extended fields of view (FOV), both of which can be useful for the design of optimized panoramic multi-camera capture devices. [Figure 8] 1 shows an image sensor with a sensor mount having an adjuster. [Figure 9] 1 shows a cross-sectional view of an improved optomechanical structure for a multi-camera capture device and a 3D view of its camera channels. [Figure 10] A cross-sectional view of two adjacent stepped edge angle compressor lenses at and near the seam and a magnified view of part (B) thereof are shown. [Figure 11] Shows 3D and top-down views of the sensor area, masking, and optical fiducials. [Figure 12] Baffles are shown. [Figure 13A] 10 shows a cross section of an improved lens design showing a multi-compressor lens group that may support the FRU concept. [Figure 13B] Another improved lens design is shown for an offset device center. [Figure 14A] 3D view of a multi-camera capture device with fins. [Figure 14B] 1 shows a multi-camera capture device with pins. [Figure 15] 1 shows an electronic system diagram for a multi-camera capture device. [Figure 16] 10A-10C show different views of an outer compressor lens element with a lens datum and an associated lens housing with a datum feature. [Figure 17] 1 shows a cross-sectional view of a portion of an assembly of an improved multi-camera panoramic image capture device, and some details thereof; [Figure 18A] 1 shows a series of views of components and features of a lens housing and channel loading support that can be used during assembly of an improved multi-camera panoramic image capture device. [Figure 18B-1] 11 shows a cross-sectional view of an alternative version of the lens housing and its interaction at or near the seam to that shown in FIG. 10. [Figure 18B-2] 18B shows a cross-sectional view of an alternative version of the inter-channel datum to that shown in FIG. 18A. [Figure 19] 18 shows a view of an alternative design to that of FIG. 10 or FIG. 17 for attachment to the central support of the camera channel. [Figure 20] 13 shows a view of another alternative design for mounting the camera channel to the central support. [Figure 21] 1 shows an alternative configuration for an improved multi-camera projection device. DETAILED DESCRIPTION OF THE INVENTION

[0008] As commonly understood in the field of optics, a lens or lens assembly typically comprises a system or device having multiple lens elements mounted within a lens barrel or housing and working together to generate an optical image. An imaging lens captures a portion of light coming from an object or objects residing in object space at some distance or distances from the lens system. The imaging lens can then form images of these objects at an output "plane," with the images having a finite size determined by magnification, as determined by the focal length of the imaging lens and the conjugate distances to the object(s) and image plane for that focal length. The amount of image light passing through the lens, from object to image, is largely determined by the size of the imaging lens's aperture stop, which is typically quantified by one or more values ​​for numerical aperture (NA) or F-number (F# or F / #).

[0009] The image quality provided by an imaging lens is determined by many characteristics of the lens design, including the choice of optical materials used in the design, the size, shape (or curvature) and thickness of the lens elements, the relative spacing of the lens elements from one another, the spectral bandwidth, polarization, the optical load (power or flux) of the transmitted light, optical diffraction or scattering, and / or lens manufacturing tolerances or errors. Image quality is typically described or quantified in terms of lens aberrations (e.g., spherical, coma, or distortion) or the relative size of the resolved spot provided by the lens, and is often also quantified by the modulation transfer function (MTF).

[0010] In a typical electronic or digital camera, an image sensor is nominally located at the imaging plane. This image sensor is typically a CCD or CMOS device that is physically attached to a heat sink or other heat removal means and also includes electronics that power the sensor and readout and communication circuitry that provides image data to data storage or image processing electronics. The image sensor typically has a color filter array (CFA), such as a Bayer filter, within the device, with the color filter pixels aligned with the image pixels to provide an RGB (red, green, blue) pixel array. Alternative filter array patterns can alternatively be used, including CYGM (cyan, yellow, green, magenta) filters or RGBW filter arrays (W=white).

[0011] In typical usage, many digital cameras are used by people or remote systems in relative isolation to capture images or photographs of a scene without any dependency or interaction with any other camera devices. In some cases, such as surveillance or security, camera operation can be directed by people or algorithms based on image content seen from another camera that has already captured overlapping, adjacent, or nearby image content. In another example, people capture panoramic images of a scene, such as a landscape scene, with an extended or wide-angle FOV by sequentially capturing a series of adjacent images while manually or automatically moving or panning to assemble the adjacent images. Image processing software, such as Photoshop or Lightroom, can then be used to stitch, mosaice, or tile the adjacent images together to represent a larger, extended scene. Image stitching or photo stitching is the process of combining multiple photographic images with overlapping fields of view to generate a segmented panoramic or high-resolution image. Image quality improvements, including exposure or color correction, can also be applied either in real time, or in a post-processing or image rendering phase, or a combination thereof.

[0012] Unless objects in a scene are directionally illuminated and / or have directional optical responses (e.g., through reflection), the available light is plenoptic, meaning that there is light traveling in all directions, or approximately so, within a given space or environment. A camera can then sample a subset of this light as image light and use it to provide a generated image showing a given view or perspective of different objects in the scene at one or more times. If the camera is moved to a different nearby location and used to capture another image of that same scene portion, both the apparent perspective and relative positions of the objects change. In the latter case, one object may now partially occlude another, while a previously hidden object becomes at least partially visible. These differences in the apparent positions or orientations of objects are known as parallax. Specifically, parallax is the displacement or difference in the apparent positions of objects viewed along two different lines of sight, measured by the angle or half-angle of inclination between these two lines.

[0013] In stereoscopic image capture or projection systems, dual-view parallax is a cue, along with shading, occlusion, and perspective, that can provide the perception of depth. For example, in a stereoscopic (3D) projection system, pairs of polarized or spectrally encoded images are projected superimposed on a screen and can be viewed by audience members wearing appropriate glasses. The amount of parallax may have an optimal range, outside of which the resulting perception of depth is either too small to be practically noticed by audience members or too large to be properly fused by the human visual system.

[0014] On the other hand, in panoramic image capture applications, parallax differences can be considered an error that can complicate both the stitching and appearance of the images. In the example of manually capturing a series of individual panoramic landscape images, if objects in the scene are sufficiently far apart (e.g., optically infinity), visual differences in perspective or parallax across the images may be too small to notice. Panoramic capture devices integrated with rotating or multi-cameras have the potential to continuously capture real-time image data at high resolution without relying on the uncertainty of manual capture. However, such devices may also introduce errors, including their own binocular parallax, image artifacts, or those of parallax, perspective, and exposure. While the resulting images can often be successfully stitched together using image processing algorithms, input image errors complicate image processing, lengthening processing times, and sometimes leaving visually apparent residual errors.

[0015] To provide context, FIG. 1 illustrates an improved integrated panoramic multi-camera capture device 100 with two adjacent cameras 120 within a housing 130 designed for reduced-parallax image capture. These cameras are alternatively referred to as image pickup units, or camera channels, or objective lens systems. Each camera 120 has multiple lens elements (see FIG. 2) mounted within a lens barrel or housing 130. Adjacent outer lens elements 137 have adjacent beveled edges 132, positioning one camera channel in close proximity to another, but which may not be touching and are therefore separated by a gap or seam 160 of finite width. A portion of the available light (□), or light rays 110, from the scene or object space 105 enters the camera 120 and becomes image light that is captured within a constrained FOV and directed to the image plane, while other light rays do not reach the camera at all. Some light rays 110 propagate into the camera and pass through the lens elements forming as field edge chief rays 170, or marginal rays, while other light rays can potentially propagate through the lens elements, creating stray or ghost light and false bright spots or images. As an example, some light rays (167) incident at a large angle on the outer surface of outer lens element 137 can travel complex paths through the camera's lens elements and create ghost images that are detectable at image plane 150.

[0016] More specifically, FIG. 2A shows a cross section of a portion of a camera 120 having a set of lens elements 135 mounted within a housing (130, not shown) within a portion of an integrated panoramic multi-camera capture device 100. A fan of light rays 110 from object space 105 diverges from on-axis to the off-axis full-field chief ray range and strikes outer lens element 137, where it is refracted and transmitted inward. Refracted and transmitted through further inner lens element 140 and aperture stop 145, this image light 115 converges to a focused image at or near image plane 150, where an image sensor (not shown) is typically located. Lens system 120 of FIG. 2A can also be defined as having a lens format consisting of outer lens element 137, or compressor lens element, and inner lens element 140, the latter of which can also be defined as consisting of a wide-angle lens group before the stop (pre-stop) and an eyepiece-like lens group after the stop (post-stop). This compressor lens element (137) directs the image light 115 sharply inward, compressing the light and allowing the overall lens assembly to provide a short focal length, while also helping to allow the camera lens housing or barrel the room necessary to provide the mechanical features necessary both to hold or mount the lens elements and to properly interact with the barrel or housing of an adjacent camera. Image light passing through the camera lens assembly from the outer lens element 137 to the image plane 150 provides an image having image quality quantifiable by image resolution, image contrast, depth of focus, and other attributes, defined by the optical aberrations (e.g., astigmatism, distortion, or spherical) and chromatic or spectral aberrations encountered by the passing light at each of the lens elements (137, 140) in the camera 120. Figure 2B shows a chief ray 170, or fan of marginal rays, incident along or near the beveled edge 132 of the outer lens element 137 of the camera optics (120) shown in Figure 2A. FIG. 2B also shows a portion of the captured polygonal or asymmetric FOV 125 extending from the optical axis 185 to a line coincident with the edge ray.

[0017] 2A , outer lens element 137 functions as a compressor lens element that redirects passing image light 115 toward second lens element 142, which is the first lens element in a group of inner lens elements 140. In this design, second lens element 142 has a very concave shape, much like the outer lens elements used in fisheye-type imaging lenses. This compressor lens element serves to sharply direct image light 115 inward, or bend the light rays, allowing the overall lens assembly to provide a short focal length, while also allowing the camera lens housing 130 or barrel the room necessary to provide the mechanical features necessary both to hold or mount lens element 135 and to properly interact with an adjacent camera barrel or housing. However, with good lens and optomechanical design, and appropriate sensor selection, camera 120 can be designed with a lens assembly that supports image resolutions of 20-30 pixels / degree, even 110 pixels / degree or more, depending on the application and device configuration.

[0018] The resulting image quality from these cameras is determined by light scattered at surfaces or within lens elements, as well as light reflected or transmitted by each lens surface. Surface transmittance and camera lens system efficiency can be improved by the use of anti-reflection (AR) coatings. Image quality can also be determined by the results of non-image light. Referring back to FIG. 1, another portion of the available light may be primarily reflected by the outer lens element 137. Still other light entering the camera 120 can be blocked or absorbed by a blackened area (not shown) provided at or near the aperture stop, the inner lens barrel surface, the edges of the lens elements, internal baffles or light-trapping features, the field stop, or some combination of other surfaces. Still other light entering the camera can also become stray or ghost light potentially visible at the image plane.

[0019] The overall image quality obtained by multiple adjacent cameras 120 in an improved integrated panoramic multi-camera capture device 100 (e.g., FIG. 1) may also depend on various other factors, including camera-to-camera variations in focal length and / or track length, as well as the magnification provided by the individual cameras. These parameters may vary depending on factors including variations in glass refractive index, variations in lens element thickness and curvature, and variations in lens element mounting. As an example, images that are tiled or mosaiced together from multiple adjacent cameras typically need to be rectified, one after the other, to correct for image size changes due to camera magnification differences (e.g., ±2%).

[0020] The images produced by the multiple cameras in the integrated panoramic multi-camera capture device 100 may vary in image quality and in other respects that result in mosaicing or tiling of the images. Specifically, the directional pointing or collection of image light through the lens elements to the image sensor of any given camera 120 may be affected by the camera's angularly distorted or asymmetric FOV.

number

[0021] In contrast to the system of FIG. 1, in a typical commercially available panoramic camera, the seams between cameras are complete gaps that can be 30-50 mm wide or even larger. Specifically, as shown in FIG. 3, a panoramic multi-camera capture device 101 may have adjacent cameras 120 or camera channels separated by large gaps or seams 160, leaving a blind spot or area 165 between them where no camera can capture an image. The actual physical seam 160 between adjacent camera channels or outer lens elements 137 (FIGS. 1 and 3) can be measured in various ways: as the actual physical distance between adjacent lens elements or lens housings, as the angular extent of the lost FOV, or as the number of "lost" pixels. However, the optical seam can be even larger, as the distance between the outer chief ray of one camera and another, due to any gaps in light reception caused by vignetting or coating limitations. For example, anti-reflection (AR) coatings are typically not applied to the edges of the optical system, and an offset margin is provided to accommodate the coated aperture (CA).

[0022] To compensate for both camera misalignment and large seams 160 and to reduce the size of the blind area 165, a typical panoramic multi-camera capture system 101 ( FIG. 3 ) has each individual camera 120 capture image light 115 from a wide FOV 125 that provides overlap 127, thereby reducing the blind area 165 and potentially lost image content. As another example, in most commercially available multi-camera capture systems 101, the gaps are 25-50+ mm wide, and the compensated FOV overlap between cameras is similarly large; for example, the portion of the FOV 125 that overlaps and is captured by two adjacent cameras 120 may be 10-50% of the cameras' FOV. The presence of such large image overlap from the shared FOV 125 wastes potential image resolution, increases image processing and image stitching time, and introduces significant image parallax and perspective errors. These errors complicate image stitching because they must be corrected or averaged out during the stitching process. In such systems, disparity is not predictable because it varies as a function of object distance. If the object distance is known, disparity can be predicted for a given field of view and inter-camera spacing. However, because the object distance is typically unknown, disparity errors result, complicating image stitching. Optical flow and general stitching algorithms determine object depth and enable image stitching, but they incur processing and time burdens.

[0023] Similarly, in a panoramic multi-camera capture device 100 of the type shown in FIG. 1 with closely integrated cameras, the width and structure of the seam 160 can be a significant factor in the operation of the overall device. However, the seam can be smaller than in FIG. 3, with the effective optical seam between the FOV edges of two adjacent cameras determined by both optical and mechanical contributions. For example, by using standard optical engineering techniques to construct lens assemblies within a housing, the mechanical width of the seam 160 between the outer lens elements 137 of adjacent cameras can be reduced to 4-6 mm. For example, it is standard practice to mount lens elements within a lens barrel or housing with a minimum radial width of 1-1.5 mm, particularly near the outermost lens element. Then, consider standard coated apertures or coating margins, as well as possible vignetting, entrance pupil aberrations, front color, and chip edges, and attempt to mount adjacent lens assemblies or housings closely together using standard techniques. Therefore, when considering both optics and mechanics, the optical seam width between adjacent lenses can easily be 8-12 mm or more.

[0024] However, with optical and optomechanical designs that allow for much smaller seams, improved versions of the panoramic multi-camera capture device (300) of the type shown in FIG. 1 are possible, with even improved parallax performance. As a first example, with regard to the present technology for improved polygonal cameras, during the initial stages of fabrication of the outer lens elements 137, these lenses may have a circular shape and be AR-coated up to or near their physical edges. If these lenses are subsequently processed to add polygonal shapes that define beveled edges 132 (e.g., FIG. 2B), the result may be that the AR coating essentially extends to the beveled lens edges. The effective optical or coated aperture can then be defined by mechanical attachment or any tolerances to standard edge polishing used in optical system fabrication to avoid edge chipping. With this approach, and a combination of other techniques described below, the optical seam can be reduced to a width of 1-5 mm.

[0025] Aspects of the present disclosure generate high-quality, low-parallax panoramic images from an improved multi-camera panoramic capture device (300), a first example portion of which is shown in FIGS. 8 and 9. This broad goal can be enabled by developing a range of design strategies to inform both optical and optomechanical lens design efforts, as well as optomechanical device design and manufacturing efforts, and strategies for improved image capture and processing. This goal can also be enabled by providing both initial and ongoing camera and device calibration. Broadly speaking, image processing, or image rendering, is a method for generating quality images from raw captured image data that depends on camera intrinsics (geometric factors such as focal length and distortion), camera extrinsics (geometric factors such as camera orientation relative to object space), other camera parameters such as vignetting and transmittance, and lighting parameters such as color and directionality. With respect to the improved multi-camera panoramic capture device 300, the use of fiducials in determining and tracking the center pixel or image centroid, exposure correction, and camera intrinsic knowledge for any given camera 320 in the device all aid in completing a reliable and repeatable tiling of images acquired from multiple adjacent cameras. Accordingly, the following description focuses broadly on providing an optical (camera or objective lens) design that can enable the desired image quality, as well as camera and device assembly approaches, managing key tolerances, camera calibration, camera intrinsic and extrinsic knowledge, and other factors that can similarly affect the resulting device performance. The improved panoramic multi-camera capture device of the present invention can be used to support a wide variety of applications or markets, including cinematic image capture, augmented reality or virtual reality (VR) image capture, surveillance or security imaging, sports or event imaging, mapping or photogrammetry, vehicle navigation, and robotics.

[0026] Before investigating optomechanical means for enabling improved panoramic multi-camera capture devices (300), means are developed for providing improved cameras 120 for use in these systems. Accordingly, the goal includes providing an improved camera (320) with both reduced parallax error and image overlap. In one aspect of this approach, the goal is to reduce residual parallax error for edge chief rays collected individually by each camera in an adjacent pair. Parallax error is defined as the change in parallax with respect to object distance (e.g., chief ray trajectories are slightly different for closer distances from the device (e.g., 3 feet) versus farther distances (e.g., 1 mile)). For example, a goal or target for reduced parallax, or effectively eliminating parallax error or "parallax-free," is that the chief rays of adjacent cameras should deviate from parallelism by ≦0.5-2.0 degrees, and preferably ≦0.01-0.1 degrees. Alternatively, or equivalently, parallax error, evaluated as a perspective error with respect to position on the imaging plane, should be reduced to ≦2 pixels, and preferably ≦0.5 pixels. As another aspect of this approach, the width of the seams 160 between adjacent cameras (e.g., 120, 320) assembled within their own lens housings should be reduced. The goal is to reduce the seam widths, both in terms of their absolute physical width and their optical or effective width. For example, the goal might be to reduce the seams 160 between adjacent outer lens elements 137 to have a maximum gap or actual physical seam in the range of only about 0.5-3.0 mm, and then reduce the maximum optical seam width to a range of only about 1-6 mm. By way of example, these reduced seam widths can translate into a reduced angular range of lost FOV of only 0.25-1.0°, or a "lost" number of pixels of only 2-20. For example, for a device that provides 8k pixels around an equirectangular image such as a 360 degree panorama, a loss of only 2-4 pixels at the seams may be acceptable, as residual image artifacts may be difficult to notice.The actual details or numerical target for substantially no parallax error or maximum optical seam width will depend on many factors, including the detailed optomechanical design of the improved camera (320) and the overall device (300), tolerance control, possible tolerances on the center offset distance or amount of extended FOV (215) and the target for low parallax, as well as the overall device specifications (e.g., diameter, sensor resolution or sensor pixels used within the imaging FOV or core FOV 205 (FIG. 7)). A further goal, enabled by some combination of the aforementioned improvements, is for each camera to reliably and quickly provide an output image from an embedded sensor package that is cropped to provide a core FOV image, and each cropped image can then be easily stitched or tiled with cropped images provided by adjacent cameras to easily provide a panoramic output image in real time from the improved multi-camera capture device (300).

[0027] An improved panoramic multi-camera capture device 300, such as those shown in FIGS. 13 and 15, may have multiple cameras arranged around a sphere to capture a 360-degree circular FOV. Alternatively, the panoramic multi-camera capture device may have multiple cameras arranged around a sphere or polyhedron shape. A polyhedron is a three-dimensional solid composed of a collection of polygons joined at their edges. One polyhedron shape, as shown in FIG. 4, is that of a dodecahedron 50, which has 12 sides or faces, each shaped as a regular pentagon 55, and 20 vertices or corners (e.g., vertex 60). A dodecahedron-shaped panoramic multi-camera capture device has cameras with pentagonal outer lens elements that image at a nominal full wide angle of 69.1°. Another shape is that of a truncated icosahedron, similar to a soccer ball, also shown in FIG. 4, which has a combination of 12 regular pentagonal sides or faces, 20 regular hexagonal sides or faces, 60 vertices, and 90 edges. More complex shapes with more sides, such as regular polyhedra, Goldberg polyhedra, or shapes with octagonal sides, or even some irregular polyhedral shapes, may be useful. For example, the Goldberg truncated dodecahedron resembles the truncated icosahedron and has both pentagonal and hexagonal facets, for a total of 42 sides. However, in general, preferred polyhedra for present purposes have hexagonal or pentagonal sides or faces, which are generally rounded and have beveled edges 132 that meet at obtuse angles. Other polyhedral shapes, such as an octahedron or regular icosahedron, can also be used, although they have triangular facets. Polyhedral facets with steeper or sharper corners, such as square or triangular facets, may be easier to manufacture compared to pentagonal and / or hexagonal facets because there are fewer edges to cut to provide polygonal edges on the outermost lens elements to define the captured polygonal FOV. However, due to their sharp corners, more care may be required during cutting, chamfering, and processing of the optics as a result. Additionally, it may be more difficult to design camera lenses and camera lens housings for optical and optomechanical performance with lens facets that have large FOVs and sharp facet angles.Typically, a 360° polygonal camera does not capture a full spherical FOV because at least a portion of one facet is sacrificed to allow for support features and power and communication cabling, such as by a mounting pole. However, if the device communicates wirelessly and is also suspended at the apex by a thin cable, the FOV lost in such physical connections can be reduced.

[0028] 1 and 2B, camera channel 120 resembles a truncated cone, or a portion thereof, where a truncated cone is a geometric solid (usually a cone or pyramid) that exists between one or two parallel planes that intersect it. In that context, the fan of chief rays 170 corresponding to the polygonal edges is refracted by outer compressor lens element 137 to nominally coincide with the edges of the truncated cone of the polyhedral geometry.

[0029] To help illustrate some of the issues related to camera geometry, FIG. 5A shows cross sections of a pentagonal lens 175 capturing a pentagonal FOV 177 and a hexagonal lens 180 capturing a hexagonal FOV 182, representing a pair of adjacent cameras whose outer lens elements have pentagonal and hexagonal shapes, such as might occur in a truncated icosahedron, or soccer-ball-shaped panoramic multi-camera capture device (e.g., 100, 300). The theoretical hexagonal FOV 182 spans a 20.9° half FOV along the side, or a 41.8° full FOV (□1), with the FOV near the vertices being even larger. The pentagonal FOV 177 supports a 36.55° FOV (□2) within a circular area, with the FOV closer to the corners or vertices being even larger. Notably, in this cross section, the pentagonal FOV 177 is asymmetric, supporting a 20 degree FOV on one side of the optical axis 185 and only a 16.5 degree FOV on the other side of the optical axis.

[0030] Optical lenses are typically designed using programs such as ZEMAX or Code V. The success of a design typically depends in part on selecting the best or most appropriate lens parameters, identified as operands, for use in a merit function. This is also true when designing lens systems for improved low-parallax multi-camera panoramic capture devices, for which there are several factors that affect performance (including, in particular, parallax) and several parameters that can be optimized individually or collectively to control it. One approach targets optimization of the "NP" point, or more significantly, a variant thereof.

[0031] By way of background, in the field of optics, there is the concept of an entrance pupil, which is the projected image of the aperture stop as seen from object space, or the virtual opening toward which image rays from object space appear to propagate before any refraction by the first lens element. By standard techniques, the location of the entrance pupil can be found by identifying the paraxial chief ray from object space 105 passing through the center of the aperture stop and projecting or extending that object space in a direction toward where it strikes the optical axis 185. In optics, an incident Gaussian or paraxial ray is understood to lie within an angular range of 10° or less from the optical axis, corresponding to a ray directed toward the center of the aperture stop, which also defines the entrance pupil location. Depending on the lens characteristics, the entrance pupil may be larger or smaller than the aperture stop and located in front of or behind the aperture stop.

[0032] In contrast, in the field of low-parallax cameras, there is the concept of a no-parallax (NP) point, or viewpoint center. Conceptually, an "NP point" is associated with a high-FOV chief ray or principal ray incident on or near the outer edge of the outermost lens element, projecting or extending its object space in a direction toward where it strikes the optical axis 185. For example, depending on the design, camera channels in a panoramic multi-camera capture device can support a half-FOV with non-paraxial chief rays at angles >31° for dodecahedron-type systems (FIG. 4) or >20° for truncated icosahedron-type systems (see FIGS. 4 and 5A). This concept of NP point projection has been applied to the design of panoramic multi-camera capture devices in conjunction with expectations for chief ray propagation and parallax control for adjacent optical systems (cameras). It can also be argued that if a camera pivots around the NP point, or multiple cameras appear to rotate around a common NP point, the parallax error is reduced and images can be aligned with little or no parallax error or perspective difference. However, in the field of low-parallax cameras, the NP point is also identified with the entrance pupil, and the axial position of the entrance pupil is estimated using the first-order optics tangent relationship between the projection of the paraxial field angle on the first lens element (see Figures 2A and 2B) and the height of the incident ray.

[0033] Thus, confusingly, in the field of low-parallax camera design, NP points have previously been associated with both the projection of the FOV chief rays' edges and the projection of chief rays that lie within the Gaussian or paraxial region. As can be seen, in practice, they both have value. Specifically, NP points associated with paraxial entrance pupils can be useful in developing initial specifications for designing and describing lenses. NP points associated with non-paraxial edge field chief rays can be useful in targeting and understanding parallax performance and in defining the conical volume or truncated cone that a lens assembly can reside within.

[0034] The projection of these non-paraxial chief rays may miss the entrance pupil defined by the paraxial chief rays due to both lens aberrations and practical geometry-related factors associated with these lens systems. Regarding the former, in well-designed lenses, image quality at the image plane is typically prioritized by limiting the impact of aberrations on resolution, telecentricity, or other attributes. Within a lens system, aberrations at intermediate surfaces, including the aperture stop, can vary widely, with emphasis on the net sum at the image plane. While aberrations at the aperture stop are often somewhat controlled to avoid vignetting, non-paraxial chief rays do not need to pass through the center of the aperture stop or the entrance pupil located along the projection paraxial axis.

[0035] To further illustrate these concepts and enable the design of improved low-parallax lens systems, note that the camera lens system 120 in FIG. 2A shows both a first NP point 190A, corresponding to the entrance pupil defined by the vector projection of a paraxial chief ray from object space 105, and an offset second NP point 190B, corresponding to the vector projection of a non-paraxial chief ray from object space. Both of these ray projections intersect the optical axis 185 at positions behind both the lens system and the image plane 150. As will be explained later, ray behavior between and within the nearby regions of projection points 190A and 190B can be complex, and neither the projection position nor the projection point has a definitive value or size. While the projection of a chief ray intersects the optical axis at a point, the projections of groups of chief rays converge toward the optical axis and intersect at different locations, which may be tightly grouped (e.g., within a few microns or tens of microns), in which case the extent or size of that “point” may depend on the set of nearby chief rays used in the analysis. On the other hand, when designing a low-parallax imaging lens for imaging a large FOV, the axial distance or difference between NP points 190A and 190B provided by the projected paraxial and non-paraxial chief rays can be significantly large (e.g., millimeters). Thus, as also explained below, the axial difference represents a useful means of parallax optimization (e.g., low-parallax budget 188) of lens systems designed for current panoramic capture devices and applications. As can also be seen, the design of the improved device (300) can be optimized to locate the device geometric center, or device center 196, outside of, but close to, this low-parallax budget 188, or alternatively, within, and preferably close to, the non-paraxial chief ray NP point.

[0036] In one embodiment, FIG. 5A shows the projection of the theoretical edge of the field of view (FOV edge 155) past the outer lens elements (lenses 175 and 180) of two adjacent cameras to provide lines pointing toward a common point (190). These lines represent the theoretical limits of complex "conical" optomechanical lens assemblies, which are typically volume-limiting pentagonal or hexagonal cones. Again, ideally, in a parallax-free multi-camera system, the entrance pupils or NP points of two adjacent cameras are co-located. However, to avoid mechanical collisions, features of a given lens assembly, including the sensor package, generally should not protrude outside the camera system's frustum cone and into the conical space of an adjacent lens assembly. However, the actual lens assemblies in a multi-camera panoramic capture device are also separated by a seam 160. Thus, the actual chief ray 170 received at the lens edge, which is inside both the mechanical seam and the physical width or aperture of the attached outer lens element (lenses 175 and 180), when projected generally toward paraxial NP point 190, may instead arrive at offset NP point 192, separated by NP point offset distance 194.

[0037] This can be better understood by considering the expanded region AA proximate to the nominal or ideal point NP 190, as shown in detail in FIG. 5B. Within the hexagonal FOV 182, rays propagating within the Gaussian or paraxial region and passing through the nominal center of the aperture stop (e.g., paraxial ray 173) can be projected to the nominal NP point 190 (corresponding to the entrance pupil) or to an offset NP point 190A at a small NP point difference or offset 193 from the nominal NP point 190. Meanwhile, the actual hexagonal lens edge chief ray 170 associated with the largest inscribed circle within the hexagon can be projected to reach a common offset NP point 192A, which may be at an even larger offset distance (194A). The two adjacent cameras in FIGS. 5A and 5B may or may not also share a co-located NP point (e.g., 190). Distance offsets can arise for a variety of reasons, including geometric considerations between the cameras (adjacent hexagonal and pentagonal cameras), geometric asymmetries within the cameras (e.g., for pentagonal cameras), or due to constraints on the practical width of the seam 160, or due to directional differences between deviating light rays.

[0038] As just mentioned, there are also potential geometric differences in the projection of incident chief rays toward a simplified nominal "NP point" (190). First, incident imaging paths from the corners or vertices or near the mid-edge (mid-chord) of a hexagonal or pentagonal lens may or may not project to a common NP point within the described range between the nominal paraxial NP point 190 and the offset NP point 192B. Also, as shown in FIG. 5B, solely due to the geometric asymmetry of the pentagonal lens, for an actual accepted FOV, associated pairs of edge chief rays 170 and 171 can project to different nominal NP points 192B that can be separated both from the paraxial NP point (190) by offset distance 194B and from each other by offset distance 194C.

[0039] Another issue is that during lens design, the best performance typically occurs on-axis or near-axis (e.g., ≤0.3 normalized field of view), near the optical axis 185. In many lenses, good imaging performance often occurs at or near the field edge by design, in which case optimization weighting is often used to enforce compliance. The worst imaging performance can then occur in the intermediate field (e.g., 0.7-0.8 of the normalized image field height). Reviewing Figures 5A and 5B again, outside the paraxial region, but not as extreme as the edge chief ray (10°<θ<20.9°), intermediate off-axis rays from the intermediate field (θ) can project toward an intermediate NP point between the nominal NP point 190 and the offset NP point 192B. However, other, more extreme, off-axis rays, specifically from the 0.7-0.8 intermediate field, which are more affected by aberrations, can project to an NP point at a location that is more or less offset from the nominal NP point 190 than the offset NP point 192B at the field edge. To accommodate differences in lens design, the non-paraxial offset "NP" point can be dropped either before (closer to the lens) the paraxial NP point (entrance pupil), as proposed in Figure 5B, or after (as shown in Figure 2A).

[0040] This is shown in more detail in Figure 5C, which essentially illustrates a more zoomed-in region AA of Figure 5B, illustrating the effects of vector projected ray paths associated with errant imaging rays converging at or near the paraxial entrance pupil (190) for an imaging lens system designed and optimized using the methods of the present approach. In Figure 5C, the projected ray paths of green errant imaging rays at multiple fields of view from the camera lens system converge within a low parallax volume 188 near one or more "NP" points. Similar illustrations of ray fans can be generated for red or blue light. Projections of paraxial rays 173 can converge at or near the nominal paraxial NP point 190, or an entrance pupil located on the nominal optical axis 185 a distance Z behind the image plane 150. Projections of field edge rays 172, including chief ray 171, converge at or near an offset NP point 192B along the optical axis 185. The NP point 192B can be quantitatively defined, for example, as the center of the plethora of full-field edge rays 172. An alternative offset NP point 192A can be identified, corresponding to the "circle of least confusion," where the paraxial, edge, and intermediate or mid-field rays converge to a minimum spot. These different "NP" points are separated from the paraxial NP point by offset distances 194A and 194B, and from each other by offset distance 194C. Thus, it can be appreciated that the convergent "NP point" for any given practical imaging lens assembly or camera lens supporting a larger paraxial FOV, or an asymmetric FOV, will typically not be a point, but instead may be an offset low-parallax (LP) smudge or quantity 188.

[0041] Within the smudge or low-parallax volume 188, various possible optimal or preferred NP points can be identified. For example, offset NP points corresponding to field edge rays 172 can be emphasized to help provide improved image tiling. Alternative intermediate-field (e.g., 0.6-0.8) NP points (not shown) can also be tracked and optimized. Additionally, the overall size and location of the “LP” smudge or volume 188, or preferred NP points therein (e.g., 192B), can be varied depending on the lens design optimization. Such parameters may also vary from lens to lens for a manufactured lens system of a given design due to manufacturing variations between lens assemblies. While FIG. 5C illustrates these alternative offset “NP points” 192A, B for non-paraxial rays as being located after the paraxial NP point 190 or further away from the lens and image plane, other lenses of this type optimized using the present approach can be provided where similar non-paraxial NP points 192A located at low-parallax volume 188 can occur between the image plane and the paraxial NP point.

[0042] FIG. 5C also shows the location relative to the center of the low-parallax multi-camera panoramic capture device, device center 196. Based on optical considerations, the improved panoramic multi-camera capture device 300 can preferably be optimized to nominally position device center 196 within low-parallax volume 188. The optimized location may include being located at or near either offset NP point 192A or 192B, or within an offset distance 194B between them, to prioritize parallax control for field edge chief rays. The actual location is then determined by parallax optimization, which can be determined by lens optimization for entrance pupil spherical aberration, direct chief ray constraints, or distortion, or a combination thereof. For example, whether spherical aberration is optimized to be over-corrected or under-corrected, and what weighting is used for the field operand in the merit function, can affect the location of the non-paraxial "NP" point for peripheral or intermediate fields. "NP" point location may also be determined by managing manufacturing tolerances and residual variations in lens system manufacturing. The device center 196 can also be positioned close to the low-parallax volume 188, but offset from it by a center offset distance 198. This approach also aids in tolerance control and can provide more space near the device center 196 for cables, circuits, cooling hardware, and related structures. In such cases, adjacent cameras 120 may consequently have offset low-parallax volumes 188 at the "NP" point (FIG. 5D) instead of being co-located (FIGS. 5A, 5B). In this example, if the device center 196 were instead positioned at or close to the paraxial entrance pupil, the NP point 190, one or more of the outer lens elements 137 of the cameras 120 would be substantially smaller than normal, and the desired full FOV would not be achievable. FIG. 13B illustrates a possible positioning of a similar lens system 920 with respect to the offset device center 910.

[0043] Thus, the no-parallax (NP) point is a useful concept to address and can usefully inform panoramic image capture and system design, aiding in the design of low-parallax error lenses. While it is ideal, its limitations must also be understood. Given this description of NP point(s) and LP smudges, understanding ray behavior in this region is crucial in enabling improved low-parallax multi-camera panoramic capture devices (such as the example lens design shown in Figure 13A) to define appropriate parameters or operands to optimize and appropriate target levels of performance to strive for. In the latter case, for example, a low-parallax lens with a track length of 65-70 mm can be designed for LP smudges as wide as 10 mm (e.g., offset distance 194A). However, alternative lens designs where this parameter is further improved may have low-parallax volume 188 with a longitudinal LP smudge width or width along the optical axis (offset 194A) of a few millimeters or less.

[0044] The width and location of the low-parallax volume 188, as well as the vector directions of projection of the various chief rays and their NP point locations within the low-parallax volume, can be controlled during lens optimization by methods using operands associated with fans of chief rays 170 (e.g., FIGS. 2A and 2B). However, the LP smudge or LP volume 188 in FIG. 5C can also be understood as a visualization of the transverse component of the spherical aberration of the entrance pupil, and this parameter can be used in an alternative, but equivalent, design optimization method to using chief ray fans. Specifically, during lens optimization, using, for example, Code V, a lens designer can create a special user-defined function or operand for the transverse component of the spherical aberration of the entrance pupil (e.g., ray height), which can then be used in a variety of ways. For example, the operand value can be calculated as the residual sum of squares (RSS) of values ​​over the entire FOV or a local field of view, using either uniform or non-uniform weighting for the field of view operand. In the latter case of local field of view preference, a value can be calculated for a position at or near the entrance pupil, or somewhere within the low-parallax volume 188, depending on a preference for paraxial, intermediate, or peripheral vision. An equivalent operand could be the width of the circle of least confusion at a plane, such as the plane of offset NP point 192A or the plane of offset NP 192B, as shown in FIG. 5C. The optimization operand can also be calculated with weighting to reduce or limit parallax error across the field of view non-uniformly, with disproportionate weighting favoring peripheral or edge vision over intermediate vision. Alternatively, the optimization operand can be calculated using weighting to provide nominally low parallax error in a nominally uniform manner across the entire field of view (within or across the core FOV 205, as in FIG. 7). That type of optimization can be particularly useful for mapping applications.

[0045] Whether a low-parallax lens design and optimization method uses operands based on chief ray or entrance pupil spherical aberration, the resulting data can also be analyzed with respect to changes in imaging perspective. Specifically, parallax errors for field of view and color can also be analyzed using calculations of the center of perspective (COP), a parameter more directly related to visible image artifacts that are low-parallax quantities, and can be evaluated in terms of image pixel error or difference for imaging objects at two different distances from the camera system. Center of perspective error is essentially the change in chief ray trajectory given multiple object distances—such as an object at close range (3 ft) versus another object at “infinity.”

[0046] In drawing and architecture, perspective is the technique of depicting solid objects on a two-dimensional surface to give a correct impression of their height, width, depth, and location in relation to one another when viewed from a particular point. For example, for drawings using linear or point perspective, objects appear smaller as their distance from the observer increases. Such illustrated objects are also subject to foreshortening, meaning that an object's dimensions along the line of sight appear shorter than its dimensions across the line of sight. Perspective works by representing light passing from the scene through an imaginary rectangle (realized as the plane of the drawing) to the viewer's eye, as if the viewer were looking through a window and depicting what they see directly on the windowpane.

[0047] Perspective is related to both parallax and stereoscopic perception. In stereoscopic image capture or projection using a pair of adjacent optical systems, perspective is a visual cue, along with dual-view parallax, shadowing, and occlusion, that can provide the perception of depth. As previously mentioned, parallax is the visual perception that an object's position or orientation appears different when viewed from different positions. In the case of image capture by a pair of adjacent cameras with at least partially overlapping fields of view, parallax image differences are a cue to stereoscopic perception or an error for panoramic imaging assemblies.

[0048] To capture an image using an optical system, whether a camera or the human eye, the geometry and performance of the optical system affect the usefulness of the resulting image for low-parallax (panoramic) or high-parallax (stereo) perception. Specifically, for an ideal lens, all chief rays from object space point precisely toward the center of the entrance pupil, which coincides with the center of perspective (COP) or center of view for the resulting image. For such an ideal lens, there are no errors in perspective or parallax.

[0049] However, with respect to real lenses, there may be residual parallax error due to both physical and image quality constraints. As previously mentioned, with respect to real lenses, the projections of paraxial chief rays from the first lens element point toward a common point, the entrance pupil, whose location can be determined as an axial distance from the front surface of that first element. On the other hand, with respect to real lenses that capture a FOV large enough to include non-paraxial chief rays, the chief rays in object space may point toward a common location or a nearby amount, but typically toward a location offset from the center of the entrance pupil. While these chief rays do not inherently coincide at a single point, they can be directed through a small, low-parallax volume 188 (e.g., an LP "smudge") with appropriate lens optimization. The longitudinal or axial variation of the rays within the LP smudge can be determined from the position of the chief rays across the optical axis. Ray error can also be measured as the lateral width or axial position of the chief rays within the LP smudge.

[0050] The concept of parallax correction is illustrated in FIG. 5D with respect to the perspective center. First camera lens 120A collects light from object space 105, including light from two outer ray fans 179A and 179B, whose chief ray projections converge toward low-parallax volume 188A, and images it into at least one core FOV. These ray fans may correspond to groups 172 of field-of-view or field-edge rays near the field of view, as shown in FIG. 2B or FIG. 5C. As shown in FIG. 5C, vector projections of such rays from object space, generally directed toward image space, within LP volume 188, favor field-edge rays, so they can cross optical axis 185 beyond the image plane at or near alternative NP point 192B, which can be selected or preferred. However, as also shown in FIG. 5C, such field-edge rays 172 need not intersect optical axis 185 at exactly the same point. These differences, when transformed back into object space 105, translate into small differences in parallax or perspective for the imaging ray bundle or fan within or across the imaging FOV of the camera lens (e.g., core FOV 205, as in FIG. 7).

[0051] A second, adjacent camera lens 120B, shown in FIG. 5D , can provide similar performance, imaging a fan of chief rays 170 from within the core FOV 205 with vector projections of those chief rays converging within a corresponding low-parallax volume 188B, including ray fan 179C. LP volumes 188A and 188B can overlap, coincide, or be offset, depending on factors including the camera geometry and seams between adjacent cameras, lens system manufacturing tolerances and compensators, or whether device center 196 is offset from LP volume 188. The more these LP volumes 188 overlap or coincide, the more the perspective centers of the two lens systems overlap. Ray fan 179B of camera lens 120A and ray fan 179C of camera lens 120B are also nominally parallel to one another, e.g., there is no parallax error between them. However, even if the lens design allows for very little residual parallax error at the edges of the FOV, manufacturing variations between lens systems can magnify the difference.

[0052] Analytically, chief ray data from a real lens can also be expressed in terms of perspective error, including chromatic error, as a function of field of view. Perspective error can then be analyzed as the position error in an image between two objects positioned at different distances or orientations. Perspective error can depend on the choice of COP position, angle within the imaging FOV, and chromatic error. For example, it can be useful to prioritize the COP to minimize green perspective error. Perspective or parallax error can be reduced by optimizing the axial position (Dz) or width of a color within the LP quantity 188 relative to the perspective center for one or more field angles within the imaging FOV. The perspective center can also be graphed and analyzed as a family of curves for Z (axial) intercept position (distance in mm) versus field of view, for each color. Alternatively, to get a better idea of ​​what the captured image will look like, the COP can be graphed and analyzed as a family of curves for the camera system as the parallax error in image pixels, for each color, versus the field of view.

[0053] During the design of a camera lens system, a goal may be to limit the parallax error to a few pixels or less for imaging within the core FOV 205 (FIG. 7). Alternatively, it may be preferable to limit the parallax error, for example, to the outer edges of the core FOV and to extended FOV regions (if provided), particularly in the peripheral field. If the residual parallax error for a camera is thus small enough, the parallax difference, seen as the perspective error between two adjacent cameras near their shared seam 160 or in seam-related regions of the extended FOV where the imaging overlaps, may similarly be limited to a few pixels or less (e.g., ≦3-4 pixels). Depending on the lens design, device design, and application, it may be possible and preferable to further reduce the parallax error for the lens system, as measured by perspective error, to ≦0.5 pixels for the entire core FOV, the peripheral field, or both. If these residual parallax errors for each of two adjacent cameras are small enough, the images can be acquired, cropped, and easily tiled while correcting or hiding image artifacts from any remaining seams 160 or blind areas 165.

[0054] Continuing with the design of the type of panoramic camera of FIG. 1, lens optimization methods and parameter selection can be important to enable an improved low-parallax multi-camera panoramic capture device (300) with multiple adjacent cameras. A camera lens 120, or system of lens elements 135, such as that of FIG. 2A, can be used as a starting point. The camera lens has compressor lens element(s) and inner lens element 140, which can also be defined as consisting of a wide-angle lens group in front of the aperture and an eyepiece-like lens group behind the aperture. In designing such a lens to reduce parallax error, it can be useful to consider how the paraxial-to-nonparaxial chief ray fan 125 (see FIG. 2A), or edge chief ray fan 170 (see FIG. 2B), or localized collection of field edge rays 172 (see FIG. 5C) or 179A, B (see FIG. 5D) are imaged by the camera lens assembly. While it is possible to optimize a lens design by using a set of merit function operands for a collection or set of chief rays (e.g., 31 defined rays), the optimization process can become cumbersome. Alternatively, in pursuit of the design of an improved low-parallax multi-camera panoramic capture device (300), it has been determined that improved performance can also be obtained by using a reduced set of ray parameters or operands that emphasize the lateral component of spherical aberration at the entrance pupil or at a similar selected surface or location (e.g., offset NP point 192A or 192B) within the LP smudge volume 188 behind the lens system. Optimization for the lateral component of spherical aberration at an alternative non-paraxial entrance pupil can be achieved using merit function weighting that emphasizes the non-paraxial chief rays.

[0055] Alternatively, in a low-parallax multi-camera panoramic capture device, the fan 170 of chief rays incident on or near the beveled edge of the outer lens element of a camera 120 (see FIG. 2B ) should be parallel to the fan 170 of chief rays incident on or near the beveled edge 132 of the outer lens element of an adjacent camera (see FIG. 1 ). Note that the “edge” of the outer lens element 137 or compressor lens is a three-dimensional structure (see FIG. 2B ), which may have a flat edge across the thickness of the glass and is subject to manufacturing tolerances of that lens element, the entire lens assembly, and the housing 130, as well as adjacent seams 160 and their structure. The location definition of where the beveled edge is cut into the outer lens element depends on factors including material properties, front color, distortion, parallax correction, tolerances, and the extent of any extra extended FOV 215. Outer lens element 137 becomes a faceted outer lens element when beveled edges 132 are cut into the lens to create a set of polygonal edges that nominally follow a polygonal pattern (eg, pentagonal or hexagonal).

[0056] A camera system 120 having an outer lens element with a polygonal shape that captures incident light from a polygonal field of view will therefore form a polygonal image at image plane 150, with the shape of the captured polygonal field nominally matching the shape of the polygonal outer lens element. Cutting these beveled edges for a given pair of adjacent cameras can affect both the imaging and optomechanical structure at or near the intervening seam 160.

[0057] Alternatively, Figure 5E illustrates "front color," which is the difference in nominal ray path due to color versus field of view when directed toward an off-axis or edge field point. Typically, for a given field point, blue light rays are offset the farthest. As shown in Figure 5E, blue light ray 157, accepted by first lens element 137, is approximately 1 mm further out in DX than accepted red light ray 158, directed toward the same field of view. If lens element 137 is not large enough, this blue light may be clipped or blurred, resulting in color shading artifacts at or near the edge of the field of view. Front color may appear in the captured image content as a rainbow-like outline of the polygonal FOV or polygon edges of outer compressor lens element 437, which acts as a field stop for the optical system. Local color transmission differences that can cause front color-related color mottle artifacts near the image edges can arise from differential vignetting at the beveled edges of the outer compressor lens element 137, or from edge truncation at the compressor lens element 438 (FIG. 13A), or through the aperture stop 145. During lens design optimization to provide an improved camera lens (320), front color can be reduced (e.g., to DX≦0.5 mm width) as part of the chromatic correction of the lens design, including through glass selection within the compressor lens group or the entire lens design, or as a trade-off in the lateral color correction. The effect of front color on the captured image can also be optomechanically reduced by designing the improved camera lens (320) to have an extended FOV 215 (FIG. 7) and by designing the optomechanics to push a straight cut or beveled lens edge 132 at or beyond the edge of the extended FOV 215, so that residual front color occurs outside the core FOV 220. The front color artifact can then be removed during an image cropping step during image processing. The effect of front or lateral color can also be reduced by spatially distinct color correction during image processing.Alternatively, the improved camera lens (320) can have a color-dependent aperture at or near the aperture stop that can provide a larger transmission opening (diameter) for blue light than for red or green light, for example.

[0058] Optical performance at or near a seam can be understood, in part, in relation to distortion ( FIG. 6 ) and a defined set of fields of view ( FIG. 7 ). Specifically, FIG. 7 illustrates the potential set of fields of view for which potential image light can be collected by two adjacent cameras. As an example, a camera with a pentagonal outer lens element, with a seam 160 separating it from the adjacent lens or camera channel, whether associated with a dodecahedron, truncated icosahedron, or other polygonal lens camera assembly, can capture an ideal FOV 200 extending to the apex (60) or polygon edge of the frustum or conical volume within which the lens resides. However, due to various physical constraints that may arise at the seam, including the finite thickness of the lens housing, the physical aspects of the beveled lens element edges, mechanical wedges, and tolerances, a smaller core FOV 205 of transmitted image light can actually be captured. The coated opening for the outer lens element 137 should encompass at least the core FOV 205 with some margin (e.g., 0.5–1.0 mm). Since lenses can be manufactured with AR coatings before beveling, the coating can extend to the seams. The core FOV 205 can be defined as the largest low-parallax field of view that a given practical camera 120 can image. Equivalently, the core FOV 205 can be defined as a sub-FOV of a camera channel whose boundaries are nominally parallel to the boundaries of its polygonal cone (see FIGS. 5A and 5B). Ideally, with small seams 160 and proper control and calibration of FOV pointing, the nominal core FOV 205 approaches or matches the ideal FOV 200 in size.

[0059] During the camera alignment and calibration process, a series of image fiducials 210 can be established along one or more edges of the core FOV 205 to assist in image processing and image tiling or mosaicing. The resulting gap between the core FOV 205 supported by a first camera and that supported by an adjacent camera can result in a blind area 165 (FIGS. 5A and 5B). To compensate for the blind area 165 and the associated loss of image content from the scene, a camera can be designed to support an extended FOV 215, which provides sufficient extra FOV to accommodate the seam width and tolerance, or offset device center 196. As shown in FIG. 7, the extended FOV 215 can be extended sufficiently to provide overlap 127 with the edge of the adjacent camera's core FOV 205, although the extended FOV 215 can be even larger. This limited image overlap may result in a small amount of image resolution loss, parallax error, and some complexity in image processing, as discussed above with respect to FIG. 3, but it may also help reduce the apparent width of the seam and blind areas. However, as provided by the present approach, if the extra overlap FOV is modest (e.g., ≦5%) and the residual parallax error therein is sufficiently small (e.g., perspective error of ≦0.75 pixels), the image processing burden may be very minimal. Image capture up to the extended FOV 215 can also be used to enable intermediate capture steps that support camera calibration and image correction during operation of the improved panoramic multi-camera capture device 300. FIG. 7 shows an inscribed circle within one of the FOV sets, corresponding to a subset of the core FOV 205, that can be captured in all directions from that camera: the common core FOV 220. The angular width of the common core FOV 220 may be useful as a quick reference to the camera's image capacity. An alternative definition of a larger common core FOV 220 to include the entire core FOV 205 may also be useful.The dashed line (225) extending from the common core FOV 220 or core FOV 205 beyond the ideal FOV 200 to nominally include the extended FOV 215 represents the region where the lens design can support careful mapping of the chief or primary rays or control of the spherical aberration of the entrance pupil to enable low parallax error imaging and easy tiling of images captured by adjacent cameras.

[0060] To reduce parallax and improve image tiling across a seam 160 spanning the distance between two adjacent usable apertures between two adjacent cameras, it may be advantageous if image light is captured substantially straight, parallel, and commonly spaced for a finite distance. The amount of FOV overlap required to provide an extended FOV and limit blind areas can be determined by controlling the relative proximity of the entrance pupil (paraxial NP point) or an alternate preferred plane (e.g., to emphasize peripheral light) within the low-parallax amount 188 relative to the device center 196 (e.g., the center of a dodecahedron shape). The amount of extended FOV 215 is preferably 5% or less (e.g., 1.8° or more additional field of view relative to a nominal core FOV of 37.5°), thereby resulting in a camera peripheral field of view of, for example, approximately 0.85-1.05°. If spacing constraints at the device center and manufacturing tolerances are well managed, the extended FOV 215 can be reduced to 1% or less additional field of view. Within the extended FOV 215, parallax should be limited to nominal system levels while both image resolution and relative illumination remain satisfactory. Parallax optimization to reduce parallax error can use either chief ray or pupil aberration constraints, targeting optimization for the high FOV region (e.g., 0.85-1.0 FOV) or beyond to include the extra camera overlap region provided by the extended FOV 215 (e.g., Figure 7, partial FOV range of approximately 0.85-1.05).

[0061] Additionally, in enabling an improved low-parallax multi-camera panoramic capture device (300) with limited parallax error and improved image tiling, it may be important to control image distortion for image light passing at or near the edges of the outer lens element's FOV, e.g., peripheral vision. In geometric optics, distortion is the deviation from the desired state in which straight lines in a scene remain straight in the image (e.g., rectilinear projection). It is a form of optical aberration and describes how light rays from a scene are mapped to an image plane. Generally, for image capture, lens assemblies used for human viewing are advantageous in limiting image distortion to a maximum of + / - 2%. In current applications, it may also be useful to have a moderate distortion of 2% or less for tiling or combining panoramic images from images captured by adjacent cameras. For reference, in cylindrical distortion, image magnification decreases with distance from the optical axis, and the apparent effect is that of an image being mapped around a spherical (or cylindrical) surface. Fisheye lenses, often used to capture hemispherical or panoramic views, typically have this type of distortion as a way of mapping an infinitely wide object plane into a finite image area. Fisheye lens distortion (251) can deviate significantly from f-theta distortion (e.g., 15% or 90° half-width (HW) for the full field of view), but only a few percent for small fields of view (e.g., ≦30° HW). As another example, in laser printing or scanning systems, f-theta imaging lenses are often used to print images with minimal banding artifacts and image processing corrections for pixel placement. Specifically, f-theta lenses are designed with a cylindrical distortion that results in a nearly constant spot or pixel size and pixel positioning that is linear with the field of view θ (h=f*θ).

[0062] Thus, an improved low-parallax camera 320 capturing a half FOV of 35-40° or less may have fisheye distortion 251 because the distortion may be sufficiently low. However, the distortion can be more advantageously optimized for the design of an improved camera lens assembly for use in the improved low-parallax multi-camera panoramic capture device (300). As a first example, it may be advantageous to provide a camera lens assembly with localized nominal f-theta distortion 250A at or near the edge of the image field, as shown in FIG. 6. In one example, image distortion 250 peaks at about 1% at about 0.75 FOV, and the lens design is not optimized to provide f-theta distortion 250A below about 0.85 FOV. However, during the lens design process, the merit function can be constrained to provide nominal f-theta-like distortion 250A or substantially flat distortion 250B for rays imaged at or near the field edge, such as for peripheral fields spanning a partial field range of about 0.9-1.0. This range of high FOV with f-theta type or flat distortion correction includes the fan of chief rays 170 or peripheral rays of FIG. 2B, including rays imaged through corners or vertices 60, such as those of a lens assembly with a hexagonal or pentagonal outer lens element 137. Additionally, due to manufacturing tolerances and dynamic effects (e.g., temperature changes) applicable to the camera 120, including both the lens element 135 and the housing 130, and to the collection of cameras 120 within a panoramic multi-camera capture device, it may be advantageous to extend the region of nominal f-theta or flattened distortion within the peripheral field beyond the nominal full FOV (e.g., 0.85 to 1.05). This is illustrated in FIG. 6, where the region of reduced or flattened distortion extends beyond the full FOV to approximately 1.05 FOV. In such a peripheral FOV range, it may be advantageous to limit the total distortion change to 0.5% or less. Peripheral field distortion is controlled to keep image "edges" straight within the adjacent pentagonal region. This may allow for more efficient use of pixels when tiling an image, and therefore faster image processing.

[0063] The preceding discussion discusses distortion in the classical sense, as image aberration at the image plane. However, in low-parallax cameras, this residual distortion is typically a trade-off or nominal cancellation of the contribution from the compressor lens element (137, or 437 and 438 in FIG. 13A ) for that of the collective inner lens element (140, or 440 in FIG. 13A ). Importantly, the ray redirection caused by the distortion contribution of the outer compressor lens element also affects both the imaging ray path and the projection chief ray path toward the low-parallax volume. This consequently means that for at least some low-parallax lens designs, distortion optimization can affect parallax or field edge NP point or perspective center optimization.

[0064] The definition of the peripheral field or subfield of view range (e.g., approximately 0.85-1.05, or including 5% or less of additional field of view), where parallax, distortion, relative illumination, resolution, and other performance factors are carefully optimized to support image tiling, may depend on the device and camera geometry. As an example, for a hexagonal lens and field of view, the lower limit of the peripheral field of view can be defined as approximately 0.83, and for a pentagonal lens, as approximately 0.8. While Figure 7 illustrates the case with two adjacent pentagonal outer lens elements and FOV sets, the approach of defining peripheral fields and extended FOVs to support small regions of overlapping image capture can be applied to multi-camera capture device designs with adjacent pentagonal and hexagonal cameras, or adjacent pentagonal cameras, or cameras with adjacent edges of other polygonal shapes or any shape or contour in general.

[0065] For the extended FOV 215 to be functionally useful, the nominal image formed on the image sensor corresponding to the core FOV 205 must underfill the used image area of ​​the image sensor at least sufficiently to allow the extended FOV 215 to also be imaged. This may be done to help accommodate actual variations in the manufactured lens assemblies from ideal, or to accommodate manufacturing variations in designs with offset device centers 196 and improved low-parallax multi-camera panoramic capture devices (300). However, as will be explained later, judicious mechanical design of the lens assemblies can help to limit mechanical displacement or wedging, affecting both the image fields of a given camera and the seams between cameras, reducing parallax error and FOV overlap or underlap. Similarly, compensators or fiducial adjustments of image FOV (core FOV 205) size and position, as well as image centroid tracking and shape tracking, can be helpful. Some combination of optimization of distortion and low- or zero-parallax imaging for the extended peripheral field of view, careful mechanical design to limit and correct for component and assembly variations, and the use of correction standards or compensators can provide a superior overall system solution. As a result, images captured from the cameras can be easily cropped to the expected nominal size and shape relative to the nominal core FOV 205, and images from multiple cameras can then be mosaiced or tiled together to form a panoramic image with reduced image post-processing effort. However, the extended FOV 215 should provide sufficient additional angular width (e.g., q1≦5% of FOV) to match or exceed the expected wedge or tilt angle q2 that may occur at seams (q1≧q2), if necessary.

[0066] In designing an improved imaging lens of the type that can be used in a low-parallax panoramic multi-camera capture device (100 or 300), several primary parameters can be calculated to inform the design effort. The key parameter is the target size of the truncated cone or conical volume based on the selected polygonal configuration (lens size (FOV) and lens shape (e.g., pentagonal)) and sensor package size. Other key parameters that can be estimated include the nominal position of the paraxial entrance pupil, the focal lengths of the compressor lens group and the wide-angle lens group, and the FOV seen by the wide-angle group.

[0067] However, design optimization for an improved camera lens (320) for use in an improved low-parallax panoramic multi-camera capture device (300) also depends on how numerous other lens attributes and performance metrics are prioritized. Specifically, relevant system parameters may include control of parallax or center of perspective (COP) error at the edge or inner field locations of the image field, or both, as optimized using spherical aberration of the chief ray fan or entrance pupil. These parameters are closely related to other important parameters, including the width and location of the "LP smudge" or volume 188, the size of any center offset distance between the entrance pupil or LP smudge and the device center 196, the target width of the gap or seam, the extent of the blind region 165, and the size of any peripheral or extended FOV to provide overlap. Relevant performance metrics may include image resolution or MTF, distortion (especially in the peripheral field and of the first compressor lens element and compressor lens group), lateral color, relative illumination, front color and color vignetting, telecentricity, and ghosting. Other relevant design variables may include mechanical and material parameters such as the number of compressor lens elements, the configuration of the compressor lens group, the wide-angle lens group, and the fisheye lens group, glass selection, the maximum allowable size of the first compressor or outer lens element, sensor package size, track length, the nominal distance from the image plane to the nearest front lens element (e.g., working distance), the nominal distance from the image plane to the entrance pupil, the nominal distance from the image plane or entrance pupil to the polygon center or device center, manufacturing tolerances and limitations, and the use of compensators.

[0068] FIG. 13A provides a cross-sectional view of an alternative and improved optomechanical design for an improved camera 320 that can be used in an improved panoramic multi-camera capture device 300. In this exemplary design, the camera lens 320 has a lens format in which the compressor lens element is divided into compressor lens groups, including first and second compressor lens elements (437 and 438). The inner lens element 400 includes a wide-angle lens group positioned in front of the aperture stop 445, which includes a fourth lens element 442, and an eyepiece lens group behind the stop. As previously mentioned, the lens system provides a paraxial NP point 490 that is offset from a non-paraxial chief ray NP point 492 located within the low-parallax volume 488. The size or width of this volume and the location of potential NP points of interest therein (e.g., paraxial, intermediate field, peripheral field, circle of least confusion-based) are determined by design priorities and parallax optimization (e.g., entrance pupil spherical aberration, chief ray fan, distortion). Although the lens formats of the example camera lens system designs of Figures 2A and 13A are similar, the lenses vary in detail and performance, including with their different compressor lens configurations. Due to differences in specifications and optimization methods and priorities, these lenses also differ in cost, performance, and manufacturability.

[0069] Depending on priorities, these lens systems can be further optimized, and various variations in lens formats may be better suited for different markets or applications. Typically, the outermost lens element, or first compressor lens element, uses Ohara SLAH52, SLAH53, and SLAH63 glasses (or equivalent glasses from Schott Glass, e.g., N-LAF36 and N-LASF43), which are high-index, low-dispersion flint glasses with a visible spectrum refractive index n of approximately 1.80 and an Abbe number Vd of approximately 41.5. It should be understood that other optical materials can be used for the lens elements in camera lens 520 in general, including the compressor lens element. For example, the use of a high-index, low-dispersion, mid-crown glass such as Ohara SLAL-18 can be useful for color correction. As another example, lens elements can also be made from optical ceramics such as Alon (n ~ 1.79, Vd ~ 57-72) or spinel, which are extremely durable materials similar to sapphire but with excellent optical transparency, low dispersion, and a controllably tunable isotropic crystal structure. It should also be understood that camera lenses of this approach can be designed with optical elements composed of or including refractive, gradient index, glass or optical polymer, reflective, aspherical or freeform, kinoform, Fresnel, diffractive or holographic, subwavelength, or metasurface optical properties. These lens systems can also be designed with achromatic or apochromatic color correction or thermal defocus desensitization.

[0070] 13A does not show how the improved camera lens might be mounted within the lens housing, but the proximity of the large outer compressor lens element 437 and the large second compressor lens element 438 that forms the doublet may require care to ensure support of these elements in close proximity. As one approach, the first compressor lens element 437 may be positionally centered on the compressor doublet (438), which in turn may be centered on a major circular datum on the inner surface of the lens housing. This datum may be a tight tolerance relative to the channel centering hub 330 to reduce tolerance buildup between adjacent camera channels.

[0071] While improving the optical design of the camera lens system is important for enabling an improved low-parallax panoramic multi-camera capture device (300), improving the optomechanical design may be equally important. As suggested above, the actual performance of the camera 120 may differ from the designed performance due to material and manufacturing variations and interactions between the individual lens elements 135 and the housing 130 and their component parts. Such variations may result in changes in image quality (e.g., aberrations, including distortion), focal length (EFL) and magnification, working distance or track length, beam pointing or imaging position, and other attributes of the camera 120. These variations may also mean that the assembly and performance of a given camera may differ from that of another camera with a nominally identical optomechanical design. For example, the focal length of a set of nominally identical cameras may vary by ±2%, which in turn causes similar variations in lens magnification and FOV. This variation can be reduced or eliminated by designing improved camera lenses to be variable-focus, including focal length compensators, such as by using lens elements whose axial position can be adjusted. Alternatively, camera 120 can be designed so that a nominal image from a nominal camera underfills the image sensor, so that an image from a camera with a larger (e.g., +2%) focal length lens also fully underfills the sensor, albeit with less margin. During calibration to determine the FOV, the EFL or magnification of the lens can be measured, and the sensor can also be aligned to be in focus with respect to that lens. Image processing software can then be used to correct the image for lens differences, including correcting image size for magnification and distortion variations between lenses.

[0072] Considering optomechanics in more detail, the axial alignment or focal position of the image sensor relative to the image plane 150 provided by the camera lens assembly (320) can be improved by appropriate mechanisms. For example, FIGS. 8 and 9 show a portion of an improved panoramic multi-camera capture device 300, in which the image sensor 270 can be assembled into a sensor package 265, which includes a plate 290, a circumferential flange, several adjustment screws 280, and flexures or springs 285. For example, three adjustment screws can be used to control X-translation coupled with Z-axis rotation, another set of three screws can be used to control Z-axis translation coupled with XY-axis rotation, and an additional screw is used to control Y-axis translation. A pair of springs can be used to hold the gimbal plate 290, simultaneously allowing X- and Z-axis translation, respectively. Other alignment designs or devices can be used within the tight space constraints allowed by the cameras 320 and the overall panoramic multi-camera capture device 300, such as using pins and micrometers or pins and shims.

[0073] Manufacturing variations for individual cameras and the optomechanical interactions between them can significantly affect the design and performance of the multi-camera capture device 300. During initial assembly of the multi-camera capture device 300, tolerances and mechanical wedge interactions between the housing 430 of a first camera 320 and the housing 430 of a second adjacent camera 320 can affect the seam 400, the aiming of the individual cameras' core FOV 205 or extended FOV 215 (causing the FOVs to overlap or underlap), and thus the FOV captured by each camera. Furthermore, the mutual alignment mounting stresses imposed on adjacent cameras by close mounting physically deform one or more of the camera housings, potentially distorting or distorting the optical imaging function of the camera lenses. As a result, the camera lens system may provide images to sensors that are shifted, rotated, or tilted out of plane. These potential problems, or the risk of their occurrence, can be exacerbated by environmental influences, such as asymmetric thermal loads or substantially asymmetric optical loads.

[0074] To combat such problems, an improved multi-camera capture device 300, as shown in FIG. 9, may include features for providing kinematic mounting of the individual cameras 320 or objective lenses. Specifically, FIG. 9 shows two views of a dodecahedral multi-camera capture device 300, including a partial cross-sectional view, in which eleven pentagonal cameras 320 are mounted on a central support 325 that occupies the nominal position of a twelfth potential camera channel. Each camera 320 has a separate base lens assembly or housing 430, consisting of a lens mount that mounts a compressor lens (437), as well as an inner lens element 440 that together comprise the base lens assembly. For each camera 320, the lens elements and housing 430 fit within a nominal conical space or volume, although they need not nominally fill that space. In practice, the sharp ray bending introduced by the compressor lens elements may mean that the inner lens elements 440 and their housings or barrels underfill the available space, and the entire lens housing 430 may taper further inward, potentially leaving an open interior volume 390 between adjacent lens assemblies.

[0075] The housing 430, or base lens assembly, in FIG. 9 also includes a turned section that can be machined on a CNC multi-axis (5-axis) machine and mates with the tripod-like channel centering hub 330. The channel centering hub 330 can be machined entirely on a lathe except for the pentagonal flange, which is completed in a finishing operation after lathe turning. Machining on a lathe means exceptional concentricity and runout can be achieved, aiding in ultimate channel alignment. The housing 430 mates with the inner diameter of the channel centering hub 330, a key part of the center-mounted mechanical assembly, which is designed to fit with it, ranging from a slip fit to a slight interference fit to ensure axial alignment without significant variation due to clearance tolerances. This same fit reduces perpendicularity error with respect to the channel axis.

[0076] The tripod or channel centering hub 330 also includes a swivel or ball pivot that engages with a socket 345 of a spherical socket array 346 provided on the central support 325. In this system, the camera 320, located at the polar position opposite the central support 325, is a precisely positioned reference channel. The central support 325 consists of a cylindrical post with a ball on top. The geometry of the central support 325 and tripod or centering hub 330 can be designed to provide more space for power and communication cables, cooling tubes, and mechanisms for securing the lens housing 430 or cables. In this example, the ball includes sockets 345, each of which can accept a ball pivot 340. The ball pivot 340 is at the end of an extension pin or ball pivot arm 342. This ball and socket portion of the central support 325 can be expensive to machine, but given the expected precision regarding socket location and depth, the advantage is that centerline pointing is controlled, while there is only one part per device 300 that requires exceptional precision. On the other hand, each of the camera channels 320 can be machined with less precision, which reduces both manufacturing and replacement costs.

[0077] Each camera lens housing 430 in FIG. 9 features an external, or outer, inter-channel datum 335 located midway along a pentagonal side. These inter-channel datums 335 can include two parallel, convex, slightly curved, protruding rods separated by a groove between them. Both datums are designed to provide single-point or localized kinematic contact or interaction between the lens housings, whereby the datum features intertwine in such a way that only one part or housing dominates from a tolerance perspective. Because they intertwine, only one part variation affects the distance between each camera channel and, therefore, the angle between the channels. Specifically, if one datum 335 is large, the other, which does not contact, dominates. Thus, only one tolerance contributes to the two parts. The intertwining of the inter-channel datums 335 from one camera 320 to another allows limited angular movement of the lens housing 430 while also limiting lateral movement between the mating (pentagonal) faces or sides.

[0078] Individually and collectively, the interaction between the camera lens housings 430 or base lens assemblies limits mechanical displacement and wedge or channel pointing error (roll, pitch, and yaw) between the cameras due to both the ball and socket arrangement and the datum features (335). Each camera channel assembly works with its neighbors to limit channel pointing error. The outer lens element 437 or the portion of the base lens assembly (430) that holds the compressor lens also has an internal functional datum that allows the compressor lens to be positioned perpendicular to the optical or mechanical channel axis, and it has additional internal datum features that limit axial misalignment. As described in more detail below with respect to FIG. 16, an additional set of alignment features provided on the polished edge of the compressor lens can also function as a datum and interact with these internal datum features to limit rotation about the pentagonal channel axis.

[0079] The use of alignment features shown in FIG. 9, specifically the ball pivot and socket datum (350 and 356) and inter-channel datum feature 335, reduces the risk of rotation, pivoting, or spreading from one camera channel (320) to another. These features therefore also help allow the joint 400 between cameras 320 to have a more consistent thickness relative to design values ​​than might otherwise occur. The use of internal features within the lens housing (e.g., compensators, adjustment screws, and shims) and external features between the lens housings (e.g., inter-channel datum, ball and socket datum, and channel loading support) helps control core or extended FOV aiming so that one camera channel can align with another adjacent channel. The combined use of inter-channel datum, ball and socket datum (FIG. 10), and channel loading support (FIG. 17) can also help reduce the device's sensitivity to mechanical or thermal loads.

[0080] 10 illustrates further design options for improving the optomechanical design of the outer lens elements 437, or compressor lens elements, proximate the seam 400 in the improved multi-camera panoramic image capture device 300. Specifically, the edges of the outer lens elements 437 can protrude above or beyond the outer edge of the lens housing 430 so that two adjacent outer lens elements 437 of adjacent cameras 320 can be in near contact at the seam 400. If these outer lens elements 437 are fabricated, at least in part, from a somewhat compliant material, some degree of actual physical contact can be possible. However, if these outer lens elements 437 are fabricated from a brittle material, such as glass, even greater care is required.

[0081] In the configuration of Figure 10, two adjacent outer lens elements 437 with protruding edges are in close contact at seam 400. In a preferred design approach, the opposing edges have a stepped edge angle 365 or structure. At the outermost portions, the two lenses and housing may result in a parallel seam 400 that is 1.0 mm or less in width.

[0082] The innermost portion, where the inner beveled edges 370 of adjacent outer lens elements 437 approach each other, can provide the lens housing 430, the inter-channel datum 335, and the flat surface datum 670. The edge of each outer lens element 437 then has a stepped edge groove 380 that can be filled with a compliant adhesive. Additionally, along the edge 432 or seam 400, the outermost edges of adjacent outer lens elements 437 extend, and these lens elements can be nearly adjacent, separated by a gap that can be only 0.5 mm wide or less. In practice, optimizing the seam width can depend on how brittle or compliant the lens material is (glass or polymer), the flexibility of the adhesive filling the seam, the use of other protective measures, and the application.

[0083] FIG. 17 provides further details regarding how these and other features can be used during alignment and assembly. Specifically, FIG. 17 shows a cross-sectional view of a portion of the improved multi-camera panoramic image capture device 300 of the present approach, in which eleven camera channels 320 are mounted on a column or vertical central support 325 through which both wiring and cooling can be provided (see FIG. 15 for further details). FIG. 17 specifically shows the design for the optomechanical hardware, although assembled without lens elements (see FIGS. 9 and 15 for illustrations of optomechanical hardware including lens elements). FIG. 18A shows in greater detail the major elements or components used in this example alignment and assembly approach.

[0084] As shown in FIG. 17 , the top camera 320 can be identified as the primary alignment channel 610, while all other camera channels 320 shown are identified as secondary channels 615. Specifically, with respect to the primary channel, ball pivot arm 342 with ball pivot 340 of channel centering hub 330 engages within ball socket datum 356 of ball socket array 346, where it interacts with locking axis retaining pin 348 that slides into a recess between ball pivot arm 342 and ball pivot 340 to prevent Z-axis (vertical) translation. Anti-rotation keying pin 349 is press-fit into a localized hole provided in the ball pivot arm to prevent rotation of the primary camera channel 610 about the z-axis. The ball pivot arm 342 of the primary camera channel 610 is lightly press-fit into its ball socket datum 356 to prevent rotation about the X and Y axes.

[0085] Alternatively, ball socket 345 may have a latching mechanism (not shown) for loading ball pivot 340 against ball socket datum 356 as a means to provide both a mechanism with reduced sensitivity to load forces and mechanical load forces being applied to the camera or device. For example, the latching mechanism may include a latch that is actuated with a linkage assembly. Compared to the use of a retaining pin, the latching mechanism is more compliant, sturdy, and reliable when camera channel 320 or device 300 are subjected to load forces. It should also be noted that lens housing 430 may also have identification markings to facilitate alignment of the housing with adjacent lens housings.

[0086] As previously shown in FIG. 9 and shown in more detail in FIG. 18A, each camera channel 320 has a pair of peripheral inter-channel datums 335 on each face, which are small (local) and nominally centered along the face or side of the lens housing. For pentagonal lenses, each of the five side faces may have a pair of datums (335). These datums have curved surfaces 373 (shown in bold and elongated for emphasis) to allow for point-to-point type contact (375). Specifically, each pair of inter-channel datums 335 has a side surface that serves as a lateral datum between the camera channels. They are curved to accommodate relative angular movement between the channels. Only each side datum has an effective single point of contact 375 where each of their mating datum radial surfaces meets. Because the radii are so large, the datum surfaces approximate a straight line. Therefore, any shift of the camera channels relative to one another, which may be an angular or centerline offset, will only have a small effect on the relative lateral offset of a given camera channel.

[0087] 17, the improved multi-camera panoramic image capture device 300 may also have channel loading supports 630 that bias all of the secondary channels 615 relative to the primary channels 610. The channel loading supports 630 employ peripheral datum pairs that are nominally identical to the inter-channel datums 335 used on each channel. The channel loading supports 630 also have spring elements 635 that facilitate loading of the secondary channels 615 relative to the primary channels 610. While the use of keyed supports may impose excessive constraints, the channel loading supports 630 may also utilize a key feature to prevent rotation.

[0088] The primary alignment channel 610 (see FIG. 17) is aligned and locked in place when its ball pivot arm 342 and ball pivot 340 engage with pins (348, 349) on the channel centering hub 330. The secondary camera channels 615 are then added and loosely aligned by fitting their ball pivots 340 against datums in sockets on the channel centering hub 330. When the channel loading support 330 is added at seam 400, it gently presses the secondary channels 615 against the primary channel 610 and against each other. The relative size of the ball socket 345 to the secondary channel's ball pivot 340 is provided such that the secondary channels 615 are constrained only in the Z direction by the ball socket 345.

[0089] Then, at seam 400, for any two faces of adjacent camera channels (610, 615) to be parallel to one another, the configuration requires that at least three camera channels have their peripheral inter-channel datums 335 contact the opposing inter-channel datum 335 at their tangent points 375. This effectively constrains the secondary channel 615 to three degrees of freedom (DOF). It has been recognized that applying traditional kinematic machine design principles, in which motion and constraints in six DOFs can be tightly limited with little crosstalk, can be inherently challenging in devices capable of arranging numerous complex polygonal faces (e.g., pentagons or hexagons) in close proximity. In cases where four or more camera channels are in close proximity, the camera channels may become overly constrained if they are not in contact with one another. The system may then be pseudo-kinematic, and mechanical stresses and strains may then cause component misalignment or damage. These potential problems can be overcome in various ways. As one example, the relatively small size and centering location of inter-channel datum 335 can limit any angular or spatial misalignment to a relatively small amount. As another example, a compliant material such as RTV can be applied to seam 400 to absorb any stress or strain, while lens housing 430 can be designed and manufactured to be sufficiently rigid to resist deformation, misalignment, or damage. Seam width variations, including dynamic ones caused by thermal or mechanical loads, can also be corrected by providing an improved camera 320 that captures image light 415 with an extended FOV 215.

[0090] As shown in FIG. 18A , the contact points can occur along a line across the radial surface of the datum. Thus, each inter-channel datum 335 makes “line” contact with the adjacent set of inter-channel datums 335 if the mating surfaces are preferably parallel; otherwise, only point contact occurs. Typically, peripheral inter-channel datums 335 make contact with the opposing surfaces at only a single point. For each inter-channel datum pair, only a single datum may make contact due to dimensional differences. This is an advantage of this design approach, since only a single datum can affect the gap or seam 400 between adjacent camera channels. This approach can be used in various designs for improved multi-camera panoramic image capture devices 300, including devices with “soccer ball” or truncated icosahedron ( FIG. 4 ) geometries, where the primary alignment (camera) channel can be an outer hexagonal lens. This approach can also be used in the construction of “hemispherical” devices (see FIG. 2 ).

[0091] FIG. 18B-1 shows a cross-sectional view across the seam of an alternative or improved version of the lens housing 430 structure and their interface near seam 400 shown in FIGS. 10 and 18A. Specifically, FIG. 18B-1 shows portions of two adjacent lens housings 430 arranged around seam 400, each housing supporting an outer compressor lens element 437 and at least one second compressor lens element 438. The lens housings 430 include inner light traps 457 and side surfaces that extend into grooves 433 cut into the edges 432 of the outer lens elements 437. The outer walls of the lens housings 430 are tapered, diverging toward the center of the device to provide greater mechanical rigidity and sturdiness. The interaction of two pairs of adjacent inter-channel datums 335 can be seen in the cross-sectional view.

[0092] FIG. 18B-2 shows an alternative cross-sectional view of these same components but cut along or within seam 400 to illustrate the interaction of inter-channel datum 335. As in FIG. 18A, adjacent inter-channel datum pairs 335a and 335b each have curved surfaces 373 that interact locally to help align or line up adjacent camera channels together. In this example, to further reduce the potential for over-constraint, the inter-channel datum pairs are asymmetric and partially offset, thus more likely to provide local point contact and less likely to cause over-constraint. Circular hole 530 provides access through the sidewall of lens housing 430 to allow for the application of adhesive or RTV used in attaching outer lens element 437 to the housing.

[0093] During assembly and alignment of the camera channel, it is also important to properly position the lens elements within the housing. For example, the compressor lens (437) may include datum features along its edge 432 that interact with a set of mating datum features on the inside of the lens housing 430. Datum features provided on the lens elements may be intelligently machined or polished features intended to limit perpendicularity and concentricity errors, or may be designed to be adhesively attached to the beveled surface of a polished glass lens. As shown in FIG. 16, these features may include flat datum features 650 fabricated at corners on the bottom surface of the compressor lens, outside the FOV. Other datums, including adjacent flat edges 660 that can be polished onto the round lens element before it is formed into a pentagon with truncated or beveled edges 370, can mate with flat surface datums 670 on the lens mount. Because glass is typically adhesively attached to metal, these flat edge datums can provide guided alignment without the risk of overconstraint. The outer lens element 437 may also include an anti-clocking datum feature 680, possibly bonded to one of the beveled surfaces.

[0094] Because of these datum features, chamfers or beveled finishes on the pentagonal lens surfaces cannot be manufactured as accurately, thus helping to reduce lens cost. Although Figure 9 shows a pentagonal-shaped compressor lens or outer lens element 437 and housing 430, these mechanical approaches to reducing misalignment errors can also be applied to hexagonal-shaped lenses, or lens elements having other polygonal shapes.

[0095] Alternatively, or in addition, the lens housing 430 may include one or more tab- or post-like structures (not shown) that protrude from the lens housing 430 outside the nominal conical space or volume and can interact with similar protruding structures on adjacent lens housings, or alternatively, with recessed structures on adjacent lens housings. For example, these protruding structures may be provided generally proximate the sensor 270 and sensor package 265, be several millimeters long, and have datum features to help kinematically control DOF, such as radial position from the device center, or tilt or rotation from one camera channel to an adjacent camera channel. A camera assembly may have two such protruding structures, symmetrically or asymmetrically positioned around the lens housing and oriented orthogonally to each other, to control different degrees of freedom. Alternatively, or in addition, the camera channel may have the lens housing 430 including one or more protruding tab or post structures positioned within the seam 400. For example, such a datum feature (not shown) could be provided in the joint at vertex 60 on the polygonal outer surface of the camera channel, protruding outside the nominal conical space or volume into the joint between two adjacent channels. Depending on the device design and intended application, the protruding tabs or structures, whether located within the outer joint 400 or recessed deeper, such as near the image sensor, can be fabricated from either a compliant or rigid material, or a combination thereof. As another alternative, one or more tabs on a given lens housing 430 need not protrude outside the nominal conical volume or truncated cone, but a clamp bridging one tab to a tab on an adjacent lens housing could provide an interface or control to limit the degrees of freedom.

[0096] FIG. 19 shows an alternative version of the camera channel interface to that of FIGS. 9 and 17 for the mounting structure near the device center. Specifically, FIG. 19 shows portions of two adjacent camera channels: upper primary channel 610 in cross section and secondary channel 615 in partial perspective view. In this case, tripod or channel centering hub 330 has socket 545, which provides a concave surface that nominally contacts a matching convex surface of center hub 550. Center hub 550 is part of a mounting mechanical assembly, nominally centered at the device center, which can be attached to support post 750. Cable 560, which has a ball on one end, can be used to pull or tension socket 545 relative to center hub 550. The primary and secondary channels can each be held together by similar tensioned cables that descend into the support posts, where they are fastened and locked. Alternatively, the primary channel 610 can be held in place with a tightened bolt (not shown). Compared to the previous approaches of Figures 9 and 17, this approach trades multiple balls and sockets for an inverted configuration with multiple sockets 545 (one per camera channel) contacting one main ball or hub 550. Tensioned cables replace previous approaches that used retaining pins, latches, or springs. The approach of Figure 19 can allow multiple camera channels to be simultaneously and reliably pulled around the center of the device in alignment with the ball hub 550. The ball hub 550 can be machined from a precision ball bearing.

[0097] It should be noted that with the assembly approach shown in Figures 9 and 17, with the primary channel 610, and secondary channel 615, and centering hub 330 interacting with the central ball socket array 346, the available space in the center of the device can be tight, within which power and communication cables, cooling pipes, and support mechanisms must fit.

[0098] As shown in FIG. 13B, an improved camera 920 having a track length 980 between the front lens center and the image plane 950 can be positioned at an offset distance 925 from the image plane 950 to a low-parallax amount 992. As one approach to improving device center crowding, the camera 920, its housing 430 (not shown), and the entire improved device 300 can be designed to provide an axial center offset distance 915 along the optical axis 985 between the low-parallax amount 992 and the device center 910, similar to that described above with respect to FIG. 5C. Designing in the offset distance 915 (e.g., 1-4 mm) can provide extra space for power, communication, or cooling connections, cables, and mechanical support for the sensor package 952 or other structural components. In the example camera system 920 shown in FIG. 13B, the improved low-parallax multi-camera panoramic capture device (300) can have a dodecahedron shape, such that the device center 910 is the center of the nominal dodecahedron polygonal structure. This offset distance interacts with the optimization of lenses near the edges of the FOV and therefore needs to be determined during the design process of the camera 920 and the overall apparatus 300. Thus, this optimization may depend on or interact with seam width, distortion correction, front color control, reduced parallax for marginal rays (edge ​​rays 972) or imaging rays 975 in general, or optimization for the range and infrastructure of LP quantity 992, lens element sizing (particularly for compressor lens group 955), or tolerances for the extended FOV 215.

[0099] As another option for providing additional access for cabling, support, and thermal management hardware, the ball-and-socket approach of FIG. 17 can be replaced with a polygonal-sided internal frame ( FIG. 20 ) with an access hole in the hollow center. For an improved multi-camera panoramic image capture device 300 constructed with a dodecahedron pattern, the internal frame can also be a dodecahedron with pentagonal faces and oriented with the internal pentagonal faces nominally aligned with the external pentagonal geometry. The internal frame can be machined separately and assembled from two or more pieces, or it can be made as a single-piece structure by casting or 3D printing. While manufacturing a single-piece frame is more complex, the resulting structure is stiffer, stronger, and can support tighter mechanical tolerances. For example, a dodecahedron frame with a hollow center can be cast from stainless steel and then selectively post-cast machined to provide precision datum features. This internal frame may then be provided with flexures or adjusters on all or most of the pentagonal faces to provide kinematic-type adjustments to reduce or avoid over-constraints during device assembly and use. As before, the adjusters available on these internal faces may be different for the secondary channels as compared to the primary channels. Alternatively, the internal frame may be made at least in part from a more compliant material, such as brass or invar. The central volume of this internal frame may be at least partially hollow, so that space can be provided for electrical cabling, thermal management hardware, and other support structures.

[0100] FIG. 20 provides an example of such an internal frame 800 with multiple pentagonal faces 810 arranged in a dodecahedron pattern with a hollow center. The internal frame 800 can be designed as a mounting mechanical assembly for an 11-camera system, with a support strut attached at the 12th position (similar to FIGS. 9 and 15). Polygonal internal frames, or half or partial internal frames, can also be used in partial or hemispherical systems, in which case a camera assembly, including an image sensor, is attached to the frame. Alternatively, a hemispherical system with internal frame 800 (see, e.g., FIG. 21) can use a central hollow space (e.g., articulation) to allow image light to pass through, including through, an intervening relay lens system (725), to reach the image sensor on the other side. As shown in FIG. 20 , one pentagonal face (810A) may have three adjusters 820, such as sets of screws or flexures, oriented nominally 120° apart, that interact with features on the camera housing and thus can be used to assist in aligning a given camera channel. As previously mentioned, the attachments and adjustments for the secondary channels may have a different design or configuration than those for the primary channel. As another alternative (not shown), one or more pentagonal faces 810A, 810B, or 810C may each include one or more adjusters that can be used to gently press the camera channel against precision V-groove structures (also not shown). These V-groove structures may be fabricated within or protrude from the inner edges of the pentagonal vertices 60 of the pentagonal faces. The internal frame approach can be used with other polyhedral device structures, such as those for icosahedrons.

[0101] FIG. 11 illustrates a convenient hardware configuration in the area of ​​the sensor 270, cover glass 272, and associated sensor package 265 (which may include electronics, cooling, and mounting). The cover glass 272 can seal or protect the sensor from the environment. The cover glass 272 can provide UV or IR cut filtering by means of a thin-film interface or dichroic coating, or that function can be provided on a separate window, external filter 295. The UV or IR cut filter reduces the level of non-visible light incident on the sensor 270, which is associated with the image light 415. Alternatively, or additionally, UV and IR cut filtering can be provided by coatings applied to lens elements, including the outer surface of the outer lens element 437 (FIG. 9). The cover glass or filter 295 can also be a UV light-absorbing glass, providing UV filtering through a combination of absorption and coating reflectance.

[0102] In providing an improved multi-camera panoramic image capture device 300 with an inner beveled edge 370 (FIGS. 10 and 16), it is recognized that centering tolerances may be poorer than typical lens elements manufactured with conventional cylindrical edges. Such decentering can consequently affect the centering of the FOV captured by the entire camera lens system 320. As one approach to correcting such errors, the positioning of the effective image centroid or central pixel can be determined either optically or electronically (FIG. 11). As another approach, the improved lens housing 430 can be designed to provide corrective lateral adjustment for one or more internal lens elements. For example, a means can be provided to adjust the positioning or tilt of an intermediate internal lens element, such as for a lens located between the aperture and the image sensor. The adjustment means can include or use a micrometer, pin, shim, flexure, or spring. Z-axis compensators for adjusting focus or magnification differences can also be provided within each camera channel using similar mechanisms. The mechanisms for enabling correction may be built into the device or camera, or may be internal, external, or a combination thereof.

[0103] While the optomechanics of FIGS. 8-10 and 16-20 can reduce alignment errors for the sensor 270, camera housing 430, and one or more cameras 320, these design improvements, and others like them, may not provide sufficient accuracy for all configurations or applications of the improved multi-camera image capture device 300. As an alternative or complementary approach, an optical reference system can be provided. Specifically, as shown in FIG. 11 , a light source 460 can direct light 462 into a window or filter 295 for optical reference within the camera 320 of the improved multi-camera panoramic image capture device 300. The optical reference light 462 can be coupled into the edge of the filter 295 and propagate by total internal reflection (TIR) ​​to an output coupler 465, which directs it toward the sensor 270, where it provides one or more illuminated spots or areas that function as optical references 475. The optical reference light 462 can be low-power infrared (IR) light, such as 785 or 835 nm, and the output coupler 465 can be one or more lenslets, prism features, or diffraction gratings. The illuminated area that serves as the optical reference 475 can be a focal spot only a few sensor pixels wide. The optical reference light 462 can be provided by a light source 460 that is mounted in a mechanically stable position relative to the image sensor 270. The reference light 462, which is light that remains within the optical substrate of the filter 295 and is directed toward the opposite edge, can be absorbed by an absorber 470, which can be provided, for example, by a black paint coating.

[0104] FIG. 11 illustrates one convenient approach for providing an optical fiducial 475 that enables FOV adjustment for the camera 320 and thus helps limit parallax error and image overlap or underlap for adjacent cameras in the improved panoramic multi-camera capture system 300. In general, however, the optical fiducial can be provided by a light source positioned proximate to the sensor. As another example, a light source can be mounted on or near the sensor surface and direct light outward toward or through the cover glass, so that the light is reflected off the optical elements and back toward the image sensor. The light source can be positioned outside the sensor's active area of ​​operation, such as a reflective coating. The reflective coating can be a localized metal or dielectric coating positioned outside the aperture used by the passing image light. As previously mentioned, the light source can then provide at least one illuminated spot of light on an active sensor pixel located proximate to, but outside, the active area of ​​operation for the image light, thereby providing the optical fiducial 475. This concept can also be extended, and the optical fiducial can be attached to or interact with other components in the system, including lens elements or lens housing structures. Thus, the relative movement of particular lens elements or subgroups thereof can be monitored to inform image tracking, trimming, or correction efforts. Additionally, if one or more lens subgroups or compensators can be actively driven, such as with a motor, the resulting data can be useful to inform these corrections.

[0105] As shown in FIG. 12 , the cameras 320 for the improved multi-camera panoramic image capture device 300 may also include a mask or internal baffle 455, which can be positioned between the outer lens element 437, or compressor lens, and the inner lens element 440 behind it. As suggested by FIGS. 9 and 16 , the beveled edges 370 of the outer lens element 437 can be fabricated through a curved outer lens surface at a set distance from and parallel to the nominal edge chief ray 170 incident along the straight pentagonal outer edge. The beveled edges 370 can have a curved bevel shape or contour. However, if the polished edges (370) have a straight bevel or chamfer, it is easier and less expensive to fabricate the outer lens elements 437 and mount them within the lens housing 430. However, this means that the volume of optical glass along the straight edges of the pentagonal (or hexagonal) outer lens element 437 that can admit light, which can then become stray light—i.e., image light that complicates image mosaicing or tiling—varies along the beveled lens edge. As provided in FIG. 12 , the baffle 455 can provide a sharp polygonal-edged opening (e.g., pentagonal or hexagonal) that follows the shape of the outer lens element 437 and core FOV 205, as well as a blackened surface to block and absorb light outside the intended FOV. Alternatively, the light-absorbing baffle 455 can be printed or coated onto the inner lens element surface. Thus, the baffle 455 or mask also defines the edges of the passing image light, thus casting an edge shadow onto the image sensor 270 (see FIG. 11 ).

[0106] The optical reference light 462 and the shadow 495 cast by the baffle 455 can be used both individually and in combination to provide an improved multi-camera capture device 300 with reduced parallax error between cameras and reduced image overlap or underlap. As shown in FIG. 11 , incident image light 415 incident on an image sensor having an active area length and width can provide an illuminated image area 450. For example, a camera 320 collecting uniform image light within a nominally pentagonal-shaped core FOV 205 can result in a pentagonal-shaped illuminated area 450 on the sensor 270. This image or illuminated area 450 may underfill the width of the sensor 270 to a greater or lesser extent, depending on the illuminated area and the sensor shape. To provide a useful datum, all or most of the projected fringe shadow 495 can define the boundary of the illuminated area of ​​active pixels within the underfilled sensor area. In some examples, the fringe shadow 495 fills most (e.g., ≥ 97%) of the active area width. The baffle 455 acts like a field stop. In a camera 320 without a baffle, the outer lens element can act like a field stop to define a pentagonal FOV.

[0107] Within the illumination area corresponding to the edge shadow 495, there may be a smaller illumination area 450 corresponding to the core FOV 205, with an intermediate extended FOV 215 between them. The extended FOV 215 may be large enough to approximately match the size of the edge shadow 495. The difference may be optimized depending on the expected mechanical alignment tolerances for the positioning of the lens assembly and baffle 455, as well as the optical sharpness or transition width of the projected shadow. Underfilling the sensor active area width allows the core FOV 205, image centroid, extended FOV 215, and edge shadow 495 to shift positions on the sensor without being clipped. During the calibration process, it may be beneficial to establish an image centroid 480 that can then be tracked over time and exposure conditions. Specifically, an initial distance or offset 485 may be determined during the calibration process and stored as a pixel count. As a result, if the illuminated image area 450 shifts during or after assembly of the camera 320 into the multi-camera capture device 300, whether due to mechanical or thermal reasons, the new location of the image centroid 480 and the shape, size, and location of the FOV 205 can be determined and compared to previously stored calibration data.

[0108] Upon completion of this process, however, the boundaries of the illumination region 450 corresponding to the desired image data must be determined. As suggested in FIG. 7, the edge shadow 495 cast by the baffle 455 can then provide a useful set of reference edges or points 210 close to, but slightly larger than, the core FOV 205. The shape and location of the shadow or occlusion cast by the baffle 455 can be determined during bench testing of a given camera 320 prior to assembly of that camera 320 into a given multi-camera capture device 300. Similar calibration data can be obtained after that assembly, and then similarly track changes in shadow positioning that may occur due to environmentally or mechanically initiated changes in the internal mounting or positioning of the lens elements (e.g., 437, 440) or the baffle 455 over time as the camera and device are used.

[0109] More specifically, the FOV edge defining baffle 455 and optical fiducials 475 (see FIGS. 11 and 12) can be used in combination to monitor or longitudinally track the position of the image centroid 480 and image field edges to assist in mosaicing or tiling of the image. Specifically, methods of projecting optical fiducials or casting optical obscurations onto the image sensor 270 can be used, either individually or in combination, to align the camera lens 320 both centrally and rotationally with respect to the image sensor 270.

[0110] Essentially, the baffle 455 or mask casts a multi-edged shadow on the sensor 270, and the shape of the baffle is sampled from around the perimeter of the projected image or illuminated area 450 captured by the sensor 270 in software or hardware. While the shadow edges are relatively sharp, they may still produce a gradient falloff, potentially extending at least a few pixels between the illuminated area 450 and the dark perimeter of the projected shadow. The edge or reference data sampled from around the edges of the occluding area and shadow gradient can then be used to derive the image centroid 480 of the lens projection. The baffle 455 or occluding shadow 495 may also have additional features to indicate and derive rotation, if desired. A calibration step can initially be used to derive the relationship of the shadow 495 to the lens center and rotation. Additionally, the dimensions and shape of the mask can be precisely measured prior to installation and compared to the size and shape of the projected shadow 495. The derived relationship of the size, shape, and center of gravity of the projected shadow 495 can accommodate expected or measured differences in the shape and size of the edge shadow 495 compared to the shape and size of the baffle. Other factors that may affect the edge shadow 495, such as mask tilt and the effect of lens distortion in the partial field of view portions (e.g., 0.85-1.01) corresponding to the outer portions of the core FOV 205 and extended FOV 215, can also be accommodated.

[0111] Additionally, as previously described, optical fiducials 475 can be projected onto unused portions of the sensor 270 using IR or visible light. The pattern formed by the fiducials can then be used to derive and calibrate the fiducials' positions to the image centroid 480, as well as the lens center and rotation. The calibration step is initially used to derive the relationship of the optical fiducials 475 to the lens center and rotation, which is then related to the distortion characteristics calculated in the calibration process. Additionally, a series of calibration images of the fiducial shadows 495 provided by the baffle 455 can be used to locate more distant features (e.g., corners) and thus confirm or determine the lens's planarity relative to the sensor's mechanical alignment. The sensor alignment or planarity can be corrected using the adjustments previously described in FIG. 8. A combined fiducial method using both the optical fiducials 475 and the projected fiducial shadows 495 has the advantage of being more robust in diverse lighting conditions, where the edges of occlusion cast methods can be inconsistent or difficult to detect.

[0112] Methods for calibrating with optical (or electronic) references and shadow shielding can be used to accurately align the modeled lens distortion derived from the calibration step for a given camera to the calibrated core FOV 205. The generated correction data can be stored in a matrix or other format in an on-board look-up table (LUT) on a local board that is part of the sensor package for each camera and sensor. When an image is captured, this correction data can be used during initial or intermediate capture steps to crop a large image corresponding to the extended FOV 215 down to an image that corresponds to the actual current core FOV 205. Similarly, other available data, such as for exposure and color correction, that can be stored locally can be applied in real time to the cropped, core FOV-sized image before it is sent to the system computer for further image processing, including stitching or tiling the images into an assembled panoramic image.

[0113] More broadly, in general, the method for calibrating optical fiducials and shadow occlusions, either in real time or during a post-processing step to correct image data captured by one or more cameras 320 of the improved multi-camera panoramic capture system 300, can be used to accurately align, for a given camera, the modeled lens distortion derived from the calibration step to the captured image. The resulting image can be accurately undistorted as needed and also corrected to accommodate other camera intrinsics (e.g., lens focal length or sensor parameters). This method can be repeated for multiple cameras 320 of the multi-camera capture system 300 to enable accurate mosaicing or tiling of multiple images together. By accurately adapting knowledge of the camera intrinsics to the captured image, the quality of mosaicing across boundaries or seams between images captured by adjacent cameras improves combined image quality and reduces errors from images that are not correctly aligned relative to the originally calculated calibrated intrinsics. This increases the speed of mosaicing or tiling, as little or no time can be spent on computationally intensive (e.g., optical flow) image feature-based methods to correct for misalignment, distortion, and lens intrinsics.

[0114] FIG. 11 also shows that portions outside the active or pixel area of ​​the rectangular image sensor 270 can be covered or shielded by a mask 455. In cases where the sensor 270 is much larger than the image illumination area 450, the use of the mask 455 prevents stray light from entering the otherwise dark areas below. As a result, these pixel values ​​are essentially always “zero” and their output can be ignored or dumped, which speeds image processing and compression times. It is also possible to replace or replace the light source 460 and optical fiducial 475 with an offset value 485 for an electronic fiducial calculated as the difference in the calibrated position of the center of gravity 480 relative to the sensor edge or mask edge. The use of an electronic fiducial may be simpler and less expensive to implement than an optical fiducial 475, but may or may not be mechanically or functionally robust, depending on the mechanical design of the camera housing 430 or the use case for the improved multi-camera capture device 300.

[0115] Differences in brightness from scene to scene or camera to camera can affect the image content incident within the illumination region 450 corresponding to the core FOV 205. Specifically, image quality and image mosaicing or tiling can vary depending on the content, such as a dark scene for one camera and a bright scene for an adjacent camera, and can affect image mosaicing and exposure levels. These differences not only make it difficult to detect image or shadow edges, but also make it difficult to determine image centroid values ​​to use during image stitching. Furthermore, these exposure variations can cause mosaiced images to appear tiled along or across edges. While either electronic or optical fiducials can be used to assist in determining image centroids and image edges, optical fiducials can also be used to enable electronic exposure control correction. Additional exposure correction can be provided by matching the centers or average values ​​of adjacent cameras and scenes to help better blend or harmonize the mosaiced images.

[0116] The mask 455 shown as part of the sensor portion of the camera 320 can be eliminated or reduced if the sensor 270 has a configuration or shape of the image illumination area 450 closer to that of the core FOV 205. Replacing the rectangular sensor with a square or nearly square alternative allows for more efficient use of available space. Specifically, only a much smaller portion of the conical volume that the camera lens assembly can nominally be designed to fit within is lost in support of the unused sensor area. The illumination area 450 can be placed on a sensor with a more appropriate active area, thus improving the effective resolution (in pixels on the sensor, or pixels per degree relative to the core FOV 205). However, the use of a square image sensor can still provide space or pixels to support the use of the optical fiducial 475. Alternatively, the rectangular image sensor can be replaced with a sensor with an active area optimized for the camera geometry, such as to have a hexagonal or pentagonal shape. Having a shape-optimized sensor can further facilitate optical design, thus making it easier to fit the camera assembly within a nominal conical volume or frustum and optimize the "NP point" location and size. Using a shape-optimized sensor can provide greater freedom to trade off or balance optical design optimization against factors including the relative distance between the image sensor and the NP point, image quality, pupil aberration, image distortion, and ray constraints for reduced parallax. As a result, optical resolution (e.g., from the lens) and angular resolution (from the sensor) can be further optimized to exceed 100 px / degree. However, the design can still underfill the shape-optimized sensor in the core FOV 205 to allow room for image capture of an extended FOV and avoid potential loss of image content due to image shift and rotation. If large enough, the underfill can also allow space for providing an optical fiducial 475.

[0117] Note that for many image sensors 270, image light is incident directly on the sensor pixels. However, many commercially available image sensors also include an integrated lenslet array that overlaps and is aligned with the pixel array. A given lenslet directs a portion of the incident image light toward the corresponding underlying pixel. This approach can reduce optical crosstalk between pixels because the incident image light is more likely to be photoelectrically converted to an electronic signal within the incident pixel than would otherwise be the case. As pixels become progressively smaller, this approach is becoming increasingly popular as a means to improve or maintain image resolution (MTF). Such sensors with integrated lenslet arrays can be used in the camera 320 of the improved multi-camera capture device 300.

[0118] Alternatively, overlapping lenslet arrays can provide an array of lenses, with any given lenslet directing light onto multiple image sensor pixels. In this case, direct image resolution may be lost, but with the benefit of gaining light field capability to provide either stereoscopic image capture or increased depth of field for focused image capture. This type of light field capability can be designed into the panoramic multi-camera capture device 300, but much of the benefit may then be lost with FOV overlap (FIG. 3). In such systems, the optical design may also have entrance pupils in the front or outer lens elements, which can cause information loss. However, incorporating light field capability into an improved panoramic multi-camera capture device 300 having multiple adjacent cameras 320 with reduced parallax error, whether from improved kinematic design, improved optomechanical design at the seam 400, or the use of electronic or optical fiducials or shadow edge masks, can improve the performance and value of the captured light field data for such systems. In particular, in such a system, a camera 320 with bright field imaging can capture a portion of the available plenoptic light to provide images with improved depth of focus and object perspective without either information loss due to parallax error or complex image fusion.

[0119] As previously mentioned, seam width can affect parallax error, the size of blind region 165, the size of any complementary image overlap (FIG. 3), and image processing and mosaicing or tiling time, so seam 400 can be an important parameter in the design of an improved multi-camera capture device 300. For example, in instances where an expensive multi-camera capture device 300 is used in a controlled environment, narrow seams with tight tolerances may be possible or acceptable. However, localized impact or stress on expensive, brittle materials (e.g., glass) can result in delamination and other damage and should be avoided. Of course, there are other external damage risks, including from external materials in contact with outer lens element 437. Therefore, because multi-camera capture device 300 can be an expensive unit, optomechanical designs that provide risk mitigation or protection for the cameras or device can be beneficial.

[0120] As one solution, seam 400 can be filled with a compliant material, such as an RTV or silicone adhesive. As another solution, the optical design of the camera lens can include an outer lens element 437, which can be designed as a polymer or plastic material, such as Zeonex, that is less brittle than glass. The use of a polymer outer lens element 437 and a compliant material-filled seam 400 can further reduce the risk.

[0121] However, in optical designs, glass typically offers better and more predictable performance than polymer materials. As one approach, the outer lens element 437 can be designed using a common, more robust, and less expensive optical glass, such as BK7. The outer lens element, whether glass or polymer, can also be overcoated with a scratch-resistant AR coating, and an oleophobic or hydrophobic coating. A compliant seam-filling adhesive should also resist penetration or contamination by water, oil, or other common materials.

[0122] It may be advantageous to design the camera lens 320 with an outer lens element 437, or lens element 435 including a compressor lens, that is more of a meniscus-type lens element than that previously suggested ( FIG. 2A ). As an example, as shown in FIG. 13A , the compressor lens is divided into a compressor lens group including a first compressor lens element (437) and a compressor lens element 438 combined as a doublet. The outer lens element 437 can then be more freely designed to have a partial meniscus-type shape while still providing both some ray or image light compression or redirection toward the wide-angle lens group including the fourth lens element 442, and primary light collection for reducing parallax. If the outer lens element is more meniscus-like or has a generally long focal length, it can be flatter and protrude less above the lens housing 430. It is therefore unlikely that light rays 410 at extreme angles from object space 405 would be accepted that could successfully reach image sensor 270 as visible or detectable ghost light, except from outside the nominal field of view 425. In such a lens design, the second compressor lens element 438, or doublet, provides additional optical power to redirect passing image light inward toward the inner fourth lens element 442.

[0123] Additionally, the camera lens 320 can be optomechanically designed so that the outer lens element 437 can be a field-replaceable unit (FRU). One way to support this approach is to design the lens element to have a more meniscus-like shape. The optomechanical design can also allow the outer lens element 437 to be mounted so that it can be easily removed and replaced if damaged. For example, the outer lens element can be mounted directly within the camera lens housing 430 using datum features such as those shown in FIG. 16, but can then be removed using one or more extraction tools. A desolvable adhesive (e.g., glyptal) or thermal adhesive with a transition temperature above the terrestrial extremes can also be used. The replacement outer lens element can then be installed in its place. Alternatively, the outer lens element can be an assembly attached to a sub-housing portion that can be removed from the main camera lens housing 430. In either case, providing a lens design where the outer lens element has a partial meniscus or long focal length design results in a design that is less sensitive to image quality and image position on the sensor, resulting in inaccurate alignment of the alternate outer lens element. Potentially resulting in cast or molded optical elements can also be used, whether made of polymer (e.g., acrylic) or glass (e.g., B270). Camera recalibration using the optical or electronic fiducials or the shadow 495 of the baffle 455 previously described further reduces the risks associated with outer lens replacement.

[0124] As another alternative to the improved multi-camera capture device 300, the entire camera 320 can be designed to be modular or potentially FRU. In such a case, the camera 320, including the housing 430, and the accompanying lens elements, including the outermost compressor lens element 437, can be removed and replaced as a unit. For example, pressure from the channel loading support 630 can be released or reduced, the camera channel's ball pivot can then be released from the ball socket 345, and a replacement camera channel can be inserted into its place. Pressure from the channel loading support 630 can then be restored. Because the relative positions of the camera channels may shift slightly during this assembly process, the process of FOV centering or calibration with a reference may then need to be repeated. Depending on the complexity of the design, FRU replacement of a modular camera-lens assembly can occur in the field, at a service center, or at the factory, but with relative ease and speed.

[0125] The improved multi-camera capture device 300, and the cameras 320 therein, may also be protected by a dome or shell (not shown) with nominally concentric inner and outer spherical surfaces through which the device can image. The addition of an outer dome can be used to enclose the generally spherical device of FIG. 15 within an interior volume, or for a generally hemispherical device such as that of FIG. 21, or for devices with alternative geometries or total FOVs. The dome can be composed of a pair of interlocking hemispherical or roughly hemispherical domes that join at a joint, or it can be a single roughly hemispherical shell (e.g., for FIG. 21). The transparent dome or shell material can be glass, plastic or polymer, hybrid or reinforced polymer material, or a robust optical material such as ceramic, sapphire, or Alon. The optically clean dome or shell helps eliminate environmental contamination and can also function as a field replaceable unit (FRU) if damaged. Replacing an FRU dome can be easier than replacing the entire camera 320 or an FRU-type outer lens element or outer lens element assembly. The dome or shell can also be enhanced with an AR, oleophobic, or hydrophobic coating on the exterior surface and an AR coating on the interior surface. The use of a dome or shell can also reduce the need for or burden of using a carrying case or shipping container, although such an enclosure can still be useful. Alternatively, or in addition, the dome or shell can be faceted to provide a series of contiguous adjacent lens elements that can serve as outer lens elements for associated adjacent camera systems 320. This approach can have the potential advantage of reducing the width of both intervening seams 160 (e.g., seam width ≦0.5 mm) and their associated blind areas 165, allowing the device center 196 to coincide with the low parallax amount 188.

[0126] As yet another alternative, the improved multi-camera capture device 300, as shown in FIGS. 14A and 14B, and the cameras 320 and their housings 430 therein, can be designed to provide protective fins 510 protruding from the seams 400 or protective struts 520 protruding from the vertices (60) or corners where multiple adjacent cameras meet. These fins 510 or struts 520 can be designed to protrude into the object space 405 but outside the captured FOV by protruding less than the extent of the blind area 165, so that they can prevent direct contact by any objects from the outside environment. These fins or struts can also be designed to be sufficiently rigid with the correct height-to-thickness aspect ratio to reduce the risk of them bending. Alternatively, the fins or struts can be designed with a somewhat compliant material to bounce back from contact forces, or a combination such as compliant fins and rigid struts can be used in tandem. The posts and fins may be supported by compliant mounts embedded within the seam 400, allowing these protective features to return toward their original designed shape during an impact event without damaging the camera 320 or multi-camera capture device 300. The use of darkened fins that protrude from the seam 400, perhaps by a few millimeters or tens of millimeters, may also provide the benefit of reducing the reception of light rays at extreme angles of incidence to the camera lens, which can result in ghost light successfully reaching the image sensor 270. The use of fins or posts protects the camera 320 and multi-camera capture device 300 from only moderate forces or impacts and may therefore be of sufficient value for some designs and applications. These fins in FIG. 14A or posts in FIG. 14B may themselves be sacrificial elements or removable FRUs. Once they have performed their job or are damaged, they can then be replaced by inserting a replacement (e.g., perhaps into a receptacle socket specially designed for that purpose).

[0127] In the previous example given, the proposed width of the seam 400 was fairly narrow. However, for clarity, in relation to the design and performance of the multi-camera capture device 300, it may be unnecessary to maintain a perfect zero separation between the beams for the chief rays 170 passing through adjacent cameras 320 and for a given object distance. Rather, the goal may be to have at most one pixel of missing information (or a few pixels, depending on the application) at the target object distance. Depending on the allowed seam width and pixel loss, the cameras 320 and multi-camera capture device 300 may be more or less robust, or more or less tolerant of both manufacturing tolerances and protective design approaches, including those described above.

[0128] Specifically, some manufacturing tolerances of the cameras 320 (see, e.g., FIG. 9 ), housing 430, outer lens element 437, and seam 400 can be tolerated or accommodated. In part, the use of optical or electronic fiducials, or shadows cast by baffles, allows for greater tolerance or flexibility in collecting low-parallax error images from adjacent cameras 320 with slight FOV overlap, as these correction approaches can enable a “centered” image to be found. However, the multi-camera capture device 300 can also tolerate some image loss. For example, for a device with a 5-foot object distance and whose cameras support a total 320,000 pixel resolution (8k output equirectangular images), a 1-pixel-wide seam 400 corresponds to a 1.2 mm gap. If the improved multi-camera capture device 300 is designed to hold mechanical seams to 3 mm after construction, the cameras can be designed for a 1.5 mm gap while allowing for some FOV overlap (□□). The device may have a mechanical design that allows, within a tolerance, a coincidence pointing error of up to □□. After camera assembly, the actual light rays collected along the edge surface shared between the camera lenses may shift, but always within the pointing error. Thus, the gap or seam 400 may be constrained to a maximum width of 3 mm. For some camera designs, markets, or applications, a wider seam may be acceptable, either in absolute size (e.g., 4.5 mm seam width) or in lost image content (e.g., 2-20 pixels per seam). The lost pixels can be corrected by increasing the FOV overlap or extended FOV 215 between adjacent cameras to capture the overlapped content, but at the cost of some parallax error and a slight increase in image processing burden. However, for a moderate amount of extended FOV (e.g., ≦5%), the residual parallax error (e.g., Figure 8B) may still be modest. If the seam 400 is smaller, and with better knowledge of the location of the image centroid 480 and image size and shape, the core FOV 205 on the sensor can be larger, and less lens performance and sensor area can be devoted to providing a larger extended FOV 215.

[0129] The seam 400 may be nominally the same width at the juncture between adjacent cameras for all cameras 320 in the improved multi-camera capture system 300. Careful camera and camera housing design, as well as inter-camera mountings (FIGS. 9 and 10), can help this occur. Nevertheless, the seam width may vary either during system assembly or during dynamic environmental conditions. For example, on one side of a first camera, the seam width between that camera and an adjacent camera may be only 0.75 pixels wide, while at the same time, the seam width between the first camera and another adjacent camera may be 3.25 pixels wide. Seam width may also vary non-uniformly. The images generated for the sensor may be shifted or rotated relative to expectations. Such variations may therefore increase parallax error for the images captured by these adjacent cameras, complicating image mosaicing or tiling. However, through the use of electronic or optical fiducials (475) or shadows cast by internal baffles 455, the image centroid and image edges (FIGS. 11 and 12) can be monitored for each camera, allowing for a quick reference to a nominal calibration or expected state. This reference or correction data can also be compared from one camera to another, with the goal of defining for each camera the effective image centroid and image edges that most effectively reduce parallax error, either individually or collectively (e.g., on average) for all cameras. This information can then be used during image processing to quickly and robustly determine image edges, allowing for efficient image mosaicing or tiling.

[0130] As noted above, with respect to FIG. 15 , a useful configuration for a multi-camera capture device 300 may be to design and manufacture a generally spherical system with multiple camera lenses distributed in a dodecahedron or truncated icosahedron arrangement. However, a problem that can arise with an improved multi-camera capture device 300 of the type shown in FIG. 5 is that with multiple camera channels and their respective image sensors confined within a nominally spherical shape, there is little room for including other components or functionality. Therefore, for some applications, a device with a generally hemispherical configuration, with potential room underneath for other hardware, may be valuable. However, because the outer lens elements and cameras are typically polygonal, a hemispherical device may have a jagged or irregular perimeter. Also, in such a system, one or more of the cameras can be designed with folds (e.g., using mirrors or prisms) so that the optical path extends through a bottom irregular peripheral surface. This configuration can provide additional room for the use of modular sensors that can be swapped in and out.

[0131] However, for a truncated icosahedron "hemispherical" version, there are six camera channels with pentagonal faces and ten camera channels with hexagonal faces, and it can be difficult for the optomechanics to provide space for so many optical path bends. FIG. 21 shows an alternative version of an improved multi-camera capture device 700 in which image light collected by each camera objective lens system 720 is directed along a nominally straight optical path, then through image relay optics (725) to a more distant image sensor located within a sensor housing 730. The original image plane provided by the objective or camera lens system is essentially an actual intermediate image plane within the larger optical system. It can be reimaged at a magnification (e.g., 1:1 or 2.75:1) to a next image plane (not shown) where the image sensor is located. Thus, advantageously, the image sensor can be larger and provide a higher pixel count, and the relay lens system 725 can refocus the image provided by the objective lens (720) at the appropriate magnification to nominally fill a more distant sensor with the projected image. Figure 21 shows a "hemisphere" above a truncated icosahedron in a semi-internal view revealing portions of two camera objective lens systems 720, each with an associated relay lens system 725 that nominally extends outside the conical space or volume. Image light from each camera 720 traverses the central volume or junction 710 as it passes through its respective relay lens system 725. The relay lens system 725 may include a field lens (not shown) behind the imaging plane of the camera lens but before the junction to help contain the space necessary for the image light to pass through. The optics of the relay lens system 725 can also be designed consistently with the camera 720 to correct or compensate for its aberrations.

[0132] To provide a hollow central volume or linkage 710 for imaging light rays from adjacent camera channels to cross each other and pass through their respective relay lens systems 725, a polygonally sided internal frame 800 (see, e.g., FIG. 20 ) has access holes to allow image light to pass through the hollow center. As previously suggested, the internal polygonal frame may have flexures or adjusters provided on all or most of the polygonal faces to provide kinematic or pseudo-kinematic adjustments and reduce or avoid over-constraints during device assembly and use. However, in this case, the hollow space or central volume of this internal frame is provided primarily to allow image light to pass through the central linkage 710 and, secondarily, may also provide room for electrical cabling and thermal management hardware. Alternatively, a mirror (not shown) may be used in the relay lens path to redirect the image light so that the overall optomechanical structure is more compact. The improved multi-camera capture device 700 also includes a support structure 740, a support pole 750, and cabling 760 for providing power and extracting signal (image) data. The support structure 740 may provide more substantial support for the camera housing 730 than is illustrated in FIG.

[0133] FIG. 15 shows an electronics system diagram for an improved multi-camera capture device 300. In this example, a dodecahedron-type device has eleven cameras 320 and an electromechanical interface at the twelfth camera position. Image data is collected from each of the eleven cameras and directed through an interface input-output module and through a cable or cable bundle to a portable computer that can provide image processing, including live image cropping and mosaicing or tiling, and camera and device control. Output image data is directed to an image display, a VR headset, or a further computer located locally or remotely. Electrical power and cooling may also be provided as needed.

[0134] Also, as previously suggested, the performance of a multi-camera capture device can be affected by both internal and external environmental factors, both in relation to the optomechanics and image quality. Each of the image sensors 270 and the entire sensor package 265, along with the data interface and power support electronics, can be a localized heat source. To reduce the thermal impact on the camera lenses and the images they provide, the mechanical design for the improved multi-camera capture device 300 can thermally isolate the sensors 270 from the lens optomechanics. To further help reduce thermal gradients between the sensors and their electronics and the optical system, micro-heat pipes or Peltier devices can be used to cool the sensors and redirect heat. Heat can be removed from the entire device by either active or passive cooling provided through the electromechanical interface at the twelfth camera position, as shown in FIG. 15. This cooling can be provided by convection or conduction (including liquid cooling), or a combination thereof.

[0135] Similarly, as previously suggested, external ambient or environmental factors can also affect the performance of a multi-camera capture device. These factors can include the effects of illuminating ambient light or thermal extremes or changes in the environment. For example, because sunlight is typically highly directional, certain outdoor image capture scenarios can result in cameras on one side of the device seeing plenoptic illumination from a brightly illuminated scene while other cameras see plenoptic illumination from a shaded scene. In such cases, the captured images may exhibit dynamic exposure changes, which can then be corrected by exposure correction, which can be provided locally (see FIG. 15 ). In an improved multi-camera capture device 300, light from the optical reference 475 can also be used for exposure correction of the captured images. Light or pixel signals from a portion of the peripheral image region between the edge of the core FOV 205 and the extended FOV 215 can also be used for exposure correction before the image is cropped to the size of the actual current core FOV 205. It should also be noted that the extended FOV 215 of a first camera may overlap, at least in part, with the extended FOV 215 of an adjacent camera, so that light level and color comparisons can be made on content or signals being captured simultaneously by both cameras. The signals or pixel data from these overlapping regions can be used to determine exposure changes between the two cameras by having a common reference point (e.g., matched feature points—using SIFT, SURF, or similar algorithms to find common feature points within the overlapping region).

[0136] Note that peripheral image or exposure data may also be retained for use in later image post-processing. Additionally, exposure correction may also be enabled by embedding photodetectors within the seams 400 or at the vertices, between the outer lens elements 437. These abrupt exposure differences may cause spatial and temporal differences in the thermal load of some image sensors 270 compared to others within the multi-camera capture device 300. The aforementioned sensor cooling, whether enabled by heat pipes, heat sinks, liquid cooling, or other means, may be designed to absorb such differences. Performance may be verified by finite element analysis (FEA).

[0137] Alternatively, one or more camera systems can be protected by the attachment of a shield or mask to cover the polygonal shape of its outer lens elements, seam-to-seam and vertex-to-vertex. Such shields can be provided to cover a single camera lens system or multiple lens systems. These shields can be shaped to generally fit the outer surface shape of the outer lens elements, and they can be used to prevent saturation or overexposure from bright directional light (e.g., sunlight) or to prevent contamination from local directional environmental factors. While these caps are nominally removable for some user applications, they can remain in use for extended periods. Excessively bright exposure from the sun or other light sources can also be controlled with an image sensor having an electronic shutter or drain, or a physical shutter or electro-optical neutral density filter, photochromic or electrochromic filter, which can be designed into, for example, the camera 320, within the grouping of inner lens elements 440. A signal to initiate or control an electronic or secondary shutter can be obtained from the image sensor or from other internal or external photodetectors. As another robustness improvement, one or more camera channels can use a dichroic color filter array integrated into the image sensor package instead of the standard dye-based CFA.

[0138] Environmental influences may also heat or cool the multi-camera capture device asymmetrically. The aforementioned nominally kinematic mounting or coupling of adjacent camera housings 430 (see FIGS. 9-10 and 17-18) to the improved multi-camera capture device 300 can help reduce this impact by attempting to deflect or average mechanical stresses and limit mechanical movement. However, it may be even more advantageous to provide channels or materials to transfer or shift asymmetric thermal loads so that they are more evenly shared between or by the cameras 320 and their housings 430. With respect to FIG. 9 , this means that the space around the lens housing 430 and channel centering hub 330, such as inner volume 390, may be filled, at least in part, with a compliant, yet highly thermally contacting, thermally conductive material (e.g., Sil-Pad (from Henkel Corporation) or CoolTherm (Lord Corporation, Cary, North Carolina, USA)) to help spatially average out asymmetrical heat loads or differences. Alternatively, or additionally, thermally conductive straps or tapes, such as the 88xx series adhesive tapes from 3M (Saint Paul, Minnesota, USA), can be used. At the same time, however, some of the effects of thermal variations on the imaging performance of the camera lens 320 can be mitigated by both judicious selection of optical glass and athermal mounting of the optical elements within the lens housing 430. In combination, an effective design approach may be to allow heat transfer or crosstalk between the lenses 320 and their housings 430 to environmental effects, while simultaneously isolating the lenses and housings from the sensors 270 and their electronics.

[0139] The improved camera 320 for use in the improved multi-camera image capture device 300 may also use an adjustable lens element to correct for thermally or mechanically induced focus changes (e.g., defocus). This adjustable lens may preferentially be located between the inner lens elements 440 and may be a liquid crystal or elastic polymer type device, such as an electrically actuated adjustable focus lens from Optotune (Dietikon, SW).

[0140] Emphasis has been placed on developing improved cameras 320 with polygonal outer lens elements to capture and image light from a polygonal FOV for use in improved multi-camera image capture devices 300. A large number of such adjacent cameras can be used in a nominally spherical or hemispherical device. However, devices 300 can be developed with fewer cameras to cover a smaller total FOV. For example, systems with only four or six low-parallax adjacent polygonal cameras may be suitable for some market applications. Additionally, a single camera with polygonal outer lens elements capturing image light with reduced parallax or perspective error from a nominally matching polygonal FOV can be used alone, such as for security or surveillance applications. For example, a single camera optomechanically designed to fit within the shape of one-eighth of an octahedron could be mounted in a corner of a room ceiling to capture image content of the room environment with little or no blind areas. Similarly, while emphasis has been placed on developing an improved camera lens system 320 in which parallax error can be reduced at least within the core FOV 205, a moderate extended FOV 215 (e.g., ≦5% excess) and image capture overlap with adjacent cameras can also be provided. Similarly, it is noted that the present approach can be extended to support possible applications with even larger overlap FOVs of image capture between adjacent cameras (e.g., 10-25% excess for a dodecahedral system with a nominal core FOV of 37.45°, or approximately 4-10° excess FOV), while simultaneously having parallax or perspective error control at least within the core FOV 205. The camera design can be extended even further to provide an even larger overlap FOV (e.g., 10-20° excess), but without the benefit of reduced parallax for angles much beyond the designed core FOV.

[0141] As mentioned above, this approach can be used to design low parallax lens systems for a variety of applications. The method for device development can be outlined, for example, as follows: 1. Define in terms of customer or application space, polygonal device configuration, resolution, expected extent of seams and blind areas, and minimum object distance. The minimum object distance is the shortest distance at which instantaneous image stitching or image merging of low-parallax overlapping images can be applied, provided image processing is acceptable. At this distance, estimate the maximum total gap or seam for which image loss or difference is generally imperceptible to the human eye. For example, for an 8k, etc. rectangular projection image, an expected image loss (along the equator) of 1 / 40% of the output image corresponds to a seam of 2 pixels wide each. Alternative minimum object distances can be defined that are even further away, at which even modest image processing is not acceptable. 2. Design the imaging lens with parallax and front color control. The design can be further modified to provide an additional or extended FOV where parallax can be controlled. This additional FOV can provide room for rainbow tinting of the remaining front color, positioned some distance outside the core FOV. 3. Design the lens and device optomechanics. Determine the expected seam width and expected thickness and wedge variations between and within seams from manufacturing and assembly tolerance variations. Determine the expected extended FOV overlap to cover the expected mechanical and optical seam width to be as high as or larger than the expected maximum optomechanical wedge and thickness variations so that parallel chief rays between adjacent lens systems can be selected without underlap (widened field of view). 4. Updated lens design to provide field of view overlap along the seams between outer compressor lens elements, controlling parallax therein while exceeding both maximum expected mechanical variation and residual front color.

[0142] In conclusion, aspects of the present disclosure provide an improved panoramic multi-camera capture device having low-parallax cameras in which optomechanics synergistically enable image capture by the cameras. The intervening mechanical seam between two adjacent cameras has a practical width and can therefore affect the extent of beam splitting or blind areas and the number of pixels lost or the value of having an extended FOV. The present application provides, in part, several means, structures, and configurations for providing robust and controlled alignment of multiple adjacent cameras in a panoramic device so that the seam width can be reliably reduced.

[0143] In order to push the chief ray to the edge of the polygonal face, the aberrations of the entrance pupil, and especially the spherical and axial chromatic aberrations of the pupil, should be optimized or reduced. For context, the entrance pupil and exit pupil, which are the projected images of the aperture stop in object space and image space, respectively, are usually academic curiosities, representing the virtual image of the aperture stop that one can see when looking through the lens.

[0144] In typical optical systems, aberrations are significant at the image plane, even if the values ​​at individual internal surfaces, whether positive or negative, are on a larger scale, with the typical goal being to provide a small net individual surface contribution to provide good image quality. On the other hand, aberrations at the aperture stop are often less of a concern, other than properly positioning the stop to define the lens system's F-stop while minimizing clipping or vignetting of the ray bundle relative to the field of view. It should be noted that when an object is placed at the aperture stop, pupil aberrations can affect the apparent image quality of the object's image as seen by a viewer at the entrance or exit pupil, although these pupil aberrations do not necessarily represent the image quality at the image plane.

[0145] In low-parallax lenses, pupil aberrations, and especially entrance pupil aberrations, are important. First, to begin properly designing a lens to control parallax, the entrance pupil must be positioned behind the image plane. Second, the redirection of peripheral chief rays from the outer compressor lens element toward the low-parallax area must be controlled. As mentioned above, optimizing the spherical aberration of the entrance pupil can be an effective way to limit parallax error. In this context, slight, moderately corrected, or insufficiently corrected optimization of the spherical aberration of the entrance pupil can occur, meaning that the nonparaxial chief ray NP point can lead to or follow the paraxial NP point, respectively. Additionally, axial chromatic aberration of the entrance pupil causes chromatic differences that can affect the optimization of the spherical aberration of the entrance pupil. Front color, which can be considered an artifact of this axial chromatic aberration, and the axial chromatic aberration itself can be reduced through the judicious selection and use of high- and low-dispersion optical materials within the lens design. While optimization of spherical aberration in the entrance pupil or chief ray pointing provides fine tuning for non-paraxial chief ray NP pointing and parallax reduction, distortion from the compressor lens group also has an increasing effect on the projected chief ray pointing for increasing fields of view, towards positions further back in the lens, and towards lower parallax amounts. The magnitude and characteristics of the distortion, which also defines the chief ray height on the outer surfaces of the outer compressor lens elements, can be significantly determined by the use of aspheric surfaces within the compressor lens group.

[0146] While this description has emphasized the design of improved multi-camera image capture devices 300 for use in broadband visible, or human-sensible, applications, these devices can also be designed for narrowband visible applications (modified using spectral filters, ultraviolet (UV), or infrared (IR) optical imaging applications). Polarizers or polarizer arrays can also be used. Additionally, while the imaging cameras 320 have been described as using an all-refractive design, the optical design can also be reflective, or catadioptric, and can use a combination of refractive and reflective optical elements.

Claims

1. A multi-camera imaging device, a plurality of cameras, each of the cameras comprising a plurality of lens elements coupled to a lens housing, the lens elements configured to collect light incident on outer lens elements of the plurality of lens elements corresponding to a core field of view as image light and direct the image light through the lens elements to provide an image at an imaging plane; A multi-camera imaging device comprising: the outer lens element and the lens housing have a common polygonal shape with multiple sides; a first camera of the plurality of cameras and a second camera of the plurality of cameras abutting each other at a pair of corresponding adjacent side surfaces of the plurality of side surfaces; a first lens housing associated with the first camera having a first datum feature along a first side, and a second lens housing associated with the second camera having a second datum feature along a second side; The first datum feature and the second datum feature contact to align the first camera and the second camera.

2. the first datum feature and the second datum feature kinematically control the position or orientation of the first lens housing and the second lens housing; the first datum feature and the second datum feature are provided both within a seam between adjacent first and second sides and near the imaging plane or image sensor of each of the first and second housings; The multi-camera imaging device according to claim 1 .

3. 2. The multi-camera imaging device of claim 1, wherein the first datum feature and / or the second datum feature comprises one or more of a curved protruding rod, a tab, a post, and a recessed structure arranged symmetrically or asymmetrically on the first housing or the second housing.

4. 2. The multi-camera imaging device of claim 1, wherein the first and second datum features comprise convexly curved bars on the outer surfaces of the respective first and second housings that can interact with equivalent datums of adjacent cameras to provide local kinematic interaction.

5. The multi-camera imaging apparatus of claim 1 , wherein the first datum feature and the second datum feature maintain a nominally constant seam width between the first side and the second side.

6. 10. The multi-camera imaging apparatus of claim 1, wherein an outer lens element associated with the first camera is mounted and aligned within the first lens housing using additional datum features provided along a beveled edge of the outer lens element associated with the first camera.

7. The multi-camera imaging apparatus of claim 6 , wherein at least one of the beveled edges has a flat edge datum that interacts with a flat surface datum of the first lens housing.

8. The multi-camera imaging device of claim 1 , wherein the first lens housing and the second lens housing are mechanically coupled to a mounting structure located proximate a center of the multi-camera imaging device.

9. 9. The multi-camera imaging device of claim 8, wherein the mounting structure comprises an internal frame having a polygonal shape that nominally matches the polygonal shape of the multi-camera imaging device, with polygonal faces that nominally match the polygonal shapes of an outer lens element associated with the first camera and an outer lens element associated with the second camera.

10. 10. The multi-camera imaging apparatus of claim 9, further comprising one or more mechanical features comprising one or more of an adjuster, a set screw, a flexure, and a v-groove that position the first lens housing onto the internal frame.

11. The multi-camera imaging apparatus of claim 8 , wherein the mounting structure comprises a camera channel centering hub.

12. The multi-camera imaging apparatus of claim 11 , wherein the first lens housing further includes a ball feature.

13. The multi-camera imaging apparatus of claim 12 , wherein the ball features engage with corresponding socket features on the camera channel centering hubs.

14. The multi-camera imaging apparatus of claim 13 , wherein the ball feature engages with a socket on the camera channel centering hub using an additional spring or latch mechanism that constrains movement of the ball feature.

15. 14. The multi-camera imaging device of claim 13, wherein the first camera is designed as a primary camera channel positioned opposite the camera channel centering hub and mounted to limit misalignment between the ball feature and the socket feature.

16. 16. The multi-camera imaging device of claim 15, wherein the second camera and a third camera of the plurality of cameras are designated as secondary camera channels, aligned with the primary camera channel, and supported directly or indirectly by a channel loading support.

17. The multi-camera imaging apparatus of claim 1 , wherein the image sensor is positioned at the image plane.

18. 10. The multi-camera imaging apparatus of claim 1, wherein the lens elements are configured to limit parallax error at or near the edges of the core field of view by limiting transverse components of spherical aberration at a plane at or near an entrance pupil.

19. the first camera images a first field of view comprising a first core field of view and a first extended field of view larger than the first core field of view; the second camera images a second field of view comprising a second core field of view and a second extended field of view that is larger than the second core field of view; The multi-camera imaging device according to claim 1 .

20. 20. The multi-camera imaging apparatus of claim 19, wherein a residual parallax error within the overlap of the first and second extended fields of view is no more than 2 pixels.

Images