Optomechanics of a panoramic capture device equipped with adjacent cameras

Abstract

A low-parallax imager includes multiple imaging lens elements arranged to capture adjacent fields of view. In some examples, adjacent imaging lens elements may meet at a datum feature to maintain a desired spacing that may allow for a partial overlap of the low parallax amount associated with each imaging lens element.

Description

Technical Field

[0001] Cross - Reference to Related Applications This disclosure claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 865,741, titled "Opto - Mechanics of Panoramic Capture Devices with Abutting Cameras", filed on June 24, 2019; U.S. Provisional Patent Application No. 62 / 952,973, titled "Opto - Mechanics of Panoramic Capture Devices with Abutting Cameras", filed on December 23, 2019; U.S. Provisional Patent Application No. 62 / 952,983, titled "Multi - camera Panoramic Image Capture Devices with a Faceted Dome", filed on December 23, 2019; and U.S. Provisional Patent Application No. 62 / 972,532, titled "Integrated Depth Sensing and Panoramic Camera System", filed on February 10, 2020, the entire contents of each of which are incorporated herein by reference.

[0002] This disclosure relates to a panoramic low - parallax multi - camera capture device having a plurality of adjacent abutting polygonal cameras. This disclosure also relates to the opto - mechanical design of cameras that capture incident light from a polygonal field of view to form a polygonal image.

Background Art

[0003] Panoramic cameras have substantial value due to their ability to simultaneously capture wide-angle images. The oldest such example is the fisheye lens, which is an ultra-wide-angle lens that produces strong visual distortion while capturing a wide panoramic or hemispherical image. The field of view (FOV) of a fisheye lens is typically between 100 and 180 degrees, but this approach has been extended to even wider angles, including ranges of 220 - 270°, as provided by U.S. Patent 3,524,697 by Y. Shimizu. Alternatively, there are mirror or reflective-based cameras that capture annular panoramic images, such as the system proposed by P. Greguss in U.S. Patent 4,930,864. Although these technologies have continued to evolve, it is difficult for them to provide a complete hemispherical or spherical image at the resolution and image quality that current applications are now seeking.

[0004] As another alternative, panoramic multi-camera devices in which multiple cameras are arranged around the perimeter or circumference of a sphere are becoming increasingly popular. However, in most of these systems, including those described in U.S. Patent 9,451,162 and U.S. Patent 9,911,454 to A. Van Hoff et al. of Jaunt Inc., the multiple cameras are sparsely present on the outer surface of the device. In order to capture a complete 360-degree panoramic image, including the gaps or seams between adjacent individual cameras, the cameras consequently have an expanded FOV that overlaps with each other. In some cases, up to 50% of the FOV or resolution of a camera can be used for the overlap between cameras, which also creates a substantial parallax difference between the captured images. Parallax is the visual perception that the position or orientation of an object appears different when viewed from different directions. In subsequent image processing, both the extra image overlap and the parallax difference complicate and significantly delay the effort to properly combine, tile, or stitch together acceptable images from the images captured by adjacent cameras to synthesize them.

[0005] There is also a panoramic multi-camera device in which a plurality of cameras are arranged around the circumference of a sphere or on the circumference of a sphere such that adjacent cameras are in contact along part or all of the adjacent edges. As an example, U.S. Patent 7,515,177 by K. Yoshikawa describes an imaging device having a number of adjacent image pickup units (cameras). Images are collected from cameras having overlapping fields of view in order to correct for mechanical errors.

[0006] More generally, in a multi-camera device, mechanical variations in the assembly and alignment of individual cameras, and of adjacent cameras to each other, can cause actual physical variations both to the cameras themselves and to the parallelism of the seam width and the edges of the cameras along the seam. These variations can, as a result, affect the FOV captured by individual cameras, the parallax error in the images captured by adjacent cameras, the extent of the "blind spot" in the FOV corresponding to the seam, the seam width, and the amount of image overlap that needs to be corrected. Thus, there is an opportunity to improve panoramic multi-camera devices and their low-parallax cameras, both with respect to optical and opto-mechanical design and in other respects.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17

Figure 18A

Figure 18B-1

Figure 18B-2

Figure 19

Figure 20

Figure 21

DETAILED DESCRIPTION OF THE INVENTION

[0008] As generally understood in the field of optics, a lens or lens assembly typically includes a system or device having a plurality of lens elements that are mounted within a lens barrel or housing and that work together to generate an optical image. An imaging lens captures a portion of the light coming from one or more objects that exist within an object space at a distance (s) from the lens system. The imaging lens can then form an image of these objects on an output “plane,” and the image has a finite size determined by the magnification, as determined by the focal length of the imaging lens and the conjugate distances to the object(s) and the imaging plane relative to that focal length. The amount of image light passing through the lens from the object to the image is largely determined by the size of the aperture stop of the imaging lens, which is typically quantified by one or more values for the numerical aperture (NA) or F-number (F# or F / #).

[0009] The image quality provided by an imaging lens is determined by a number of characteristics of the lens design, including the selection of the optical materials used in the design, the size, shape (or curvature) and thickness of the lens elements, the relative spacing between the lens elements, the spectral bandwidth, polarization, the optical loading (power or flux) of the light passing through, optical diffraction or scattering, and / or lens manufacturing tolerances or errors. Image quality is typically described or quantified in terms of lens aberrations (e.g., spherical, coma, or distortion), or the relative size of the resolution points provided by the lens, and is often also quantified by the modulation transfer function (MTF).

[0010] In a typical electronic or digital camera, an image sensor is nominally disposed on an imaging plane. This image sensor is typically a CCD or CMOS device, which is physically attached to a heat sink or other heat removal means, and also includes electronic equipment for supplying power to the sensor, and readout and communication circuitry for providing image data to data storage or image processing electronic equipment. The image sensor typically has a color filter array (CFA), such as a Bayer filter within the device, and positions color filter pixels in alignment with image pixels to provide an array of RGB (red, green, blue) pixels. Alternative filter array patterns, including CYGM filters (cyan, yellow, green, magenta) or RGBW filter arrays (W = white), can be used instead.

[0011] In typical usage, many digital cameras are used by people or remote systems relatively separately to capture an image or photograph of a scene without any dependency or interaction with any other camera device. In some cases, such as for surveillance or security, the operation of the camera can be directed by people or algorithms based on the image content seen from another camera that has already captured overlapping, adjacent or nearby image content. In another example, people capture a series of adjacent images continuously, either manually or while automatically moving or pivoting, to capture an extended or wide-angle FOV of a scene, such as a panoramic image of a landscape scene, by assembling adjacent images. Subsequently, image processing software, such as Photoshop or Lightroom, can be used to stitch together adjacent images, mosaic, or tile them to represent a larger extended scene. Stitching of images or photographs is a process of combining multiple photographic images with overlapping fields of view to produce a segmented panorama or high-resolution image. Image quality improvements, including exposure or color correction, can also be applied either in real-time, or in a post-processing or image rendering phase, or any combination thereof.

[0012] Unless the objects within the scene are directionally illuminated and / or do not have a directional light response (e.g., by reflection, etc.), the available light is plenoptic, meaning that in a given space or environment, the light travels in all directions or approximately so. As a result, the camera can sample a subset of this light as image light and use it to provide a generated image showing a given view or perspective of one or more objects within the scene at one or more points in time. When the camera is moved to different nearby positions to capture another image of the same part of the scene, both the apparent perspective and the relative positions of the objects change. In the latter case, one object may now partially occlude another object, while an object that was previously hidden becomes at least partially visible. These differences in the apparent position or orientation of the objects are known as parallax. Specifically, parallax is the displacement or difference in the apparent position of an object seen along two different lines of sight, and is measured by the angle or half-angle of the inclination between these two lines.

[0013] In a stereoscopic image capture or projection system, the dual-view parallax is a cue along with shading, occlusion, and perspective that can provide a sense of depth. For example, in a stereoscopic (3D) projection system, a pair of polarized or spectrally encoded images are projected overlaid on the screen and can be viewed by a viewer member wearing appropriate glasses. The amount of parallax can have an optimal range, outside of which the resulting sense of depth is either too small to be actually noticed by the viewer member or too large to be properly fused by the human visual system.

[0014] On the one hand, in panoramic image capture applications, parallax differences can be regarded as errors that can complicate both image stitching and appearance. In an example of manually capturing a series of panoramic landscape images individually, when the objects in the scene are far enough away (e.g., optically at infinity), the visual differences in perspective or parallax across the images may be too small to notice. A panoramic capture device integrated with a rotating camera or multi-camera has the potential to continuously capture real-time image data at high resolution without relying on the uncertainties of manual capture. However, such a device can also incorporate its own binocular parallax, image artifacts, or errors including that of parallax, perspective, and exposure. The resulting images can often be successfully stitched together using image processing algorithms, but the input image errors complicate the image processing and lengthen the processing time, and at the same time sometimes leave visually obvious residual errors.

[0015] To provide context, FIG. 1 shows an improved integrated panoramic multi-camera capture device 100 having two adjacent cameras 120 within a housing 130 designed for reduced parallax image capture. These cameras are alternatively referred to as image pickup units, or camera channels, or objective lens systems. Each camera 120 has a plurality of lens elements (see FIG. 2) mounted within a lens barrel or housing 130. The adjacent outer lens elements 137 have adjacent sloped edges 132 and are disposed in close proximity to, but not in contact with, one camera channel to another camera channel and are thus separated by a finite width gap or seam 160. A portion of the available light (□), or light rays 110, from the scene or object space 105 enters the cameras 120 and is captured within the constrained FOV and becomes image light directed towards the imaging surface, while other light rays do not reach the cameras at all. Some of the light rays 110 propagate within the camera and pass through lens elements configured as field edge chief rays 170, or peripheral rays, while other light rays potentially propagate through the lens elements and can give rise to stray light or ghost light and false bright spots or images. As an example, some light rays (167) that are incident at a large angle on the outer surface of the outer lens element 137 can pass through a complex path through the lens elements of the camera and give rise to ghost images detectable at the imaging surface 150.

[0016] More specifically, FIG. 2A shows a cross-section of a portion of camera 120 having a set of lens elements 135 mounted within a housing (not shown, 130) within a portion of the integrated panoramic multi-camera capture device 100. A fan of light rays 110 from object space 105 spreads over the range of chief rays from on-axis to off-axis full field of view and is incident on outer lens element 137, refracted, and transmitted inwardly. This image light 115, refracted and transmitted through further inner lens element 140 and through aperture stop 145, converges to a focused image at or near imaging plane 150, where an image sensor (not shown) is typically disposed. The lens system 120 of FIG. 2A can also be defined as having a lens form consisting of outer lens element 137 or compressor lens element, and inner lens element 140, the latter of which can also be defined as consisting of a wide-angle lens group before the stop (pre-stop), and an eyepiece-like lens group after the stop (post-stop). This compressor lens element (137) sharply directs the image light 115 inwardly, compressing the light and enabling the entire lens assembly to provide a short focal length, while at the same time allowing room necessary for the camera lens housing or barrel to provide both the mechanical features necessary to hold or attach the lens elements and to properly interact with the barrel or housing of an adjacent camera. The image light passing through the camera lens assembly from outer lens element 137 to imaging plane 150 provides an image having image quality that can be quantified by image resolution, image contrast, depth of field, and other attributes, the quality of which is defined by optical aberrations (e.g., astigmatism, distortion, or spherical) and chromatic or spectral aberrations that the passing light encounters at each of the lens elements (137, 140) within camera 120. FIG. 2B shows chief ray 170, or a fan of peripheral rays, incident along or near the tapered edge 132 of outer lens element 137 of the camera optical system (120) shown in FIG. 2A. FIG. 2B also shows a portion of the captured, polygonal or asymmetric FOV 125 that spreads from optical axis 185 to a line coinciding with the marginal ray.

[0017] In the camera lens design shown in FIG. 2A, the outer lens element 137 functions as a compressor lens element that redirects the passing image light 115 towards the second lens element 142, and the second lens element 142 is the first lens element of a group of inner lens elements 140. In this design, the second lens element 142 has exactly a concave shape, much like the outer lens elements used in fisheye type imaging lenses. This compressor lens element sharply directs the image light 115 inward or bends the light rays, enabling the entire lens assembly to provide a short focal length, while at the same time allowing the camera lens housing 130 or barrel to have the mechanical features necessary both for holding or attaching the lens element 135 and for properly interacting with the adjacent camera barrel or housing. However, using good lens and optomechanical design, as well as appropriate sensor selection, the camera 120 can be designed with a lens assembly that supports image resolutions of 20 - 30 pixels / degree, 110 pixels / degree or more, depending on the application and device configuration.

[0018] The resulting image quality from these cameras is also determined by light that scatters on the surface or within the lens elements, and light that is reflected or transmitted at each lens surface. The surface transmittance and camera lens system efficiency can be improved by the use of anti-reflection (AR) coatings. The image quality can also be determined by the results of non-image light. Referring back to FIG. 1, other portions of the available light can be mainly reflected by the outer lens element 137. Still other light entering the camera 120 can be blocked or absorbed by a blackened area (not shown) provided at or near the aperture stop, the inner lens barrel surface, the edges of the lens elements, the inner baffle or light trapping features, the field stop, or some combination of other surfaces. Still other light entering the camera can potentially become stray light or ghost light that is visible at the imaging plane.

[0019] The overall image quality obtained by a plurality of adjacent cameras 120 within an improved integrated panoramic multi-camera capture device 100 (e.g., FIG. 1) can also be determined by various other factors, including variations between cameras in focal length and / or track length, as well as the magnification provided by individual cameras. These parameters can vary depending on factors including variations in glass refractive index, variations in lens element thickness and curvature, and variations in lens element mounting. As an example, images that are tiled or mosaicked together from a plurality of adjacent cameras typically need to be corrected from one to the next in order to compensate for image size changes due to camera magnification differences (e.g., ±2%).

[0020] The images generated by the plurality of cameras within the integrated panoramic multi-camera capture device 100 can also vary in terms of image quality and other aspects that result in image mosaicking or tiling. Specifically, the directional pointing or collection of image light through the lens elements to the image sensor of any given camera 120 can vary depending on whether the camera is angularly distorted or has an asymmetric FOV

Number

[0021] Compared with the system of FIG. 1, in a typical commercially available panoramic camera, the seam between cameras can be a complete gap that is 30 - 50 mm wide, or more. Specifically, as shown in FIG. 3, the panoramic multi-camera capture device 101 can have adjacent cameras 120 or camera channels separated by a large gap or seam 160, and there is a blind spot or blind area 165 between them where no camera can capture an image. The actual physical seam 160 between adjacent camera channels or outer lens elements 137 (FIGS. 1 and 3) can be measured in various ways, as the actual physical distance between adjacent lens elements or lens housings, as the angular range of the lost FOV, or as the number of "lost" pixels. However, the optical seam can be even larger due to any gap in light reception caused by vignetting or coating limitations, as the distance between the outer chief rays of one camera and another. For example, an anti-reflection (AR) coating typically does not adhere to the edges of the optical system, and an offset margin is provided to provide a coated aperture (CA).

[0022] To correct for both camera misalignment and large seams 160, and to reduce the size of the blind region 165, a typical panoramic multi-camera capture device 101 (FIG. 3) causes each of the individual cameras 120 to capture image light 115 from a wide FOV 125 that provides an overlap 127, thereby reducing the blind region 165 and minimizing the potentially capturable image content that is lost. As another example, in most commercially available multi-camera capture devices 101, the gap is 25-50+ mm wide and the corrective FOV overlap between cameras is similarly large. For example, overlapping, the portion of the FOV 125 captured by two adjacent cameras 120 can be 10-50% of the camera's FOV. The presence of such large image overlaps from the shared FOV 125 wastes potential image resolution, increases image processing and image stitching times, and at the same time incorporates significant image parallax and perspective errors. These errors complicate image stitching because they need to be corrected or averaged during the stitching process. In such a system, the parallax varies as a function of object distance and is thus not predictable. If the object distance is known, the parallax can be predicted for a given field of view and the spacing between cameras. However, since the object distance is typically not known, the parallax error consequently complicates image stitching. Optical flow and general stitching algorithms determine object depth to enable image stitching, but involve processing power and a time burden.

[0023] Similarly, in a panoramic multi-camera capture device 100 of the type of FIG. 1 with closely integrated cameras, the width and structure at the seam 160 can be important elements in the operation of the entire device. However, the seam can be made smaller than in FIG. 3 using the effective optical seam between the FOV edges of two adjacent cameras determined by both optical and mechanical contributions. For example, by using standard optical engineering techniques to construct the lens assembly within the housing, the mechanical width of the seam 160 between the outer lens elements 137 of adjacent cameras can be reduced to 4 - 6 mm. For example, it is standard practice to mount the lens elements, specifically near the outermost lens element, within a lens barrel or housing having a minimum radial width of 1 - 1.5 mm. Then, consider standard coated apertures or coating margins, and consider possible vignetting, entrance pupil aberration, front color, chip edges, and attempt to mount adjacent lens assemblies or housings in proximity using standard techniques. Thus, considering both the optical system and the mechanism, the optical seam width between adjacent lenses can be well over 8 - 12 mm.

[0024] However, an improved version of the panoramic multi-camera capture device (300) of the type of FIG. 1 is possible with an optical and opto-mechanical design that allows for much smaller seams and has further improved parallax performance. As a first example, with respect to this technology for improved polygonal cameras, during the initial stages of manufacturing the outer lens elements 137, these lenses can have a circular shape and can be AR coated up to and near their physical edges. If these lenses are subsequently processed to add a polygon defining a beveled edge 132 (e.g., FIG. 2B), the AR coating can effectively extend up to the beveled lens edge. An effective optical or coated aperture can then be defined by any tolerances for mechanical mounting or standard edge polishing used in optical system manufacturing to avoid edge chipping. With this approach, and combinations of other techniques described later, the optical seam can be reduced to a width of 1 - 5 mm.

[0025] Aspects of the present disclosure generate high-quality low-disparity panoramic images from an improved multi-camera panoramic capture device (300), with portions of a first example thereof shown in FIGS. 8 and 9. This broad goal can be enabled by developing a system-wide design strategy for informing both optical and opto-mechanical lens design efforts, as well as opto-mechanical device design and manufacturing efforts, and strategies for improved image capture and processing. This goal can also be enabled by providing both initial and ongoing camera and device calibration. Broadly, image processing or rendering of an image is a method for generating a quality image from raw captured image data determined by camera intrinsics (geometric factors such as focal length and distortion), camera extrinsics (geometric factors such as the orientation of the camera with respect to object space), other camera parameters such as vignetting and transmittance, and lighting parameters such as color and directivity. With respect to the improved multi-camera panoramic capture device 300, the use of criteria in the determination and tracking of the center pixel or image centroid, exposure correction, and knowledge of camera intrinsics for any given camera 320 within the device are all aids towards the completion of a reliable and repeatable tiling of images acquired from multiple adjacent cameras. Accordingly, the following description is broadly focused on providing an optical (camera or objective lens) design that can enable a desired image quality, as well as camera and device assembly approaches, management of critical tolerances, camera calibration, knowledge of camera intrinsics and extrinsics, and other factors that can similarly affect the resulting device performance. The improved panoramic multi-camera capture device of the present invention can be used to support a wide variety of applications or markets, including cinematic image capture, augmented or virtual reality (VR) image capture, surveillance or security imaging, sports or event imaging, mapping or photogrammetry, vehicle navigation, and robotics.

[0026] Before investigating the optomechanical means to enable an improved panoramic multi-camera capture device (300), means are developed to provide a camera 120 that is improved for use in these systems. Accordingly, the goal includes providing an improved camera (320) having both reduced parallax error and image overlap. As one aspect of this approach, the goal is to reduce the residual parallax error for the marginal chief rays individually collected by each camera of an adjacent pair. Parallax error is defined as the change in parallax with respect to object distance (e.g., for a distance closer to the device (e.g., 3 feet), the chief ray trajectory is slightly different compared to a far distance (e.g., 1 mile)). For example, one goal or target for reduced parallax, or effectively eliminating the parallax error, or making it "parallax-free", is that the chief rays of adjacent cameras should deviate from being parallel to each other by ≤ 0.5 - 2.0 degrees, and preferably by ≤ 0.01 - 0.1 degrees. Alternatively, or equivalently, since the parallax error is evaluated as a perspective error with respect to the position on the imaging plane, it should be reduced to ≤ 2 pixels, and preferably to ≤ 0.5 pixels. As another aspect of this approach, the width of the seam 160 between adjacent cameras (e.g., 120, 320) assembled within their own lens housings should be reduced. The goal is to reduce the width of the seam with respect to both their absolute physical width and their optical width or effective width. For example, the goal is to reduce the seam 160 between adjacent outer lens elements 137 to have a maximum gap or actual physical seam within the range of about 0.5 - 3.0 mm, and then reduce the maximum optical seam width to within the range of about 1 - 6 mm. As an example, these reduced seam widths can be converted to a reduced angular range of lost FOV of 0.25 - 1.0 degrees, or a "lost" number of pixels of 2 - 20 pixels. For example, for a device that provides 8k pixels around the perimeter of a rectangular (equirectangular) image such as a 360-degree panorama, a loss of only 2 - 4 pixels at the seam may be acceptable since residual image artifacts may be difficult to notice.Without substantial parallax error, or the actual details or numerical targets of the maximum optical seam width, are determined by many factors, including the improved camera (320) and the detailed optomechanical design of the entire device (300), the management of tolerances, the possible tolerances for the center offset distance or the amount of the extended FOV (215) and the targets for its low parallax, as well as the specifications of the entire device (e.g., the diameter, the imaging FOV or the sensor resolution or sensor pixels used within the core FOV 205 (FIG. 7)). A further goal, enabled by some combination of the foregoing improvements, is that each camera provides an output image reliably and quickly from an embedded sensor package that is trimmed to provide a core FOV image, and then each trimmed image can be easily stitched or tiled with the trimmed images provided by adjacent cameras to easily provide a panoramic output image in real time from the improved multi-camera capture device (300).

[0027] An improved panoramic multi-camera capture device 300, such as that shown in FIGS. 13 and 15, may have a plurality of cameras arranged peripherally around a sphere to capture a 360-degree annular FOV. Alternatively, the panoramic multi-camera capture device may have a plurality of cameras arranged around a spherical or polyhedral shape. A polyhedron is a three-dimensional solid consisting of a collection of polygons adjacent at their edges. One polyhedral shape, as shown in FIG. 4, is that of an icosahedron 50, which has 12 sides or faces, each shaped as a regular pentagon 55, and 20 vertices or corners (e.g., vertex 60). A panoramic multi-camera capture device formed in the shape of an icosahedron has cameras with pentagonal outer lens elements that image at a nominal full angle of 69.1°. Another shape, also shown in FIG. 4, is that of a truncated icosahedron, like a soccer ball, which has a combination of 12 regular pentagonal sides or faces, 20 regular hexagonal sides or faces, 60 vertices, and 90 edges. Even more complex shapes with more sides, such as regular polyhedra, Goldberg polyhedra, or shapes with octagonal sides, or even some irregular polyhedral shapes, may be useful. For example, a Goldberg beveled icosahedron resembles a truncated icosahedron and has a total of 42 sides of both pentagonal and hexagonal facets. However, generally, the polyhedra preferred for current purposes have hexagonal or pentagonal sides or faces, which are generally rounded in shape with the beveled edges 132 intersecting at obtuse angles. Other polyhedral shapes, such as octahedra or regular icosahedra, have triangular facets and can be used. Polyhedral facets with sharper or more acute angles, such as square or triangular facets, may be easier to manufacture compared to pentagonal or hexagonal facets because they require fewer edges to be cut to provide the polygon edges on the outermost lens element to define the polygonal FOV being captured. However, due to their sharp angles, an additional level of care may be required during the cutting, beveling, and processing of the optical system as a result. Additionally, designing camera lenses and camera lens housings for optical and optomechanical performance may be even more difficult for lens facets with large FOVs and sharp facet angles.Typically, a 360° polyhedral camera does not capture a full spherical FOV because at least a portion of one facet is sacrificed to enable support features and power and communication cable routing, such as by attachment struts. However, if the device communicates wirelessly and is also suspended at the apex by a thin cable, the FOV lost due to such physical connections can be reduced.

[0028] As shown in FIGS. 1 and 2B, camera channel 120 is frustum-shaped, or part thereof, where a frustum is a geometric solid (usually a cone or pyramid) that exists between one or two parallel planes that cross it. In that context, a fan of chief rays 170 corresponding to the edges of the polygon is refracted by the outer compressor lens element 137 to nominally coincide with the edges of the frustum of the polyhedral geometry.

[0029] To assist in illustrating some issues related to the camera geometry, FIG. 5A shows a cross-section of a pentagonal lens 175 that captures a pentagonal FOV 177 and a hexagonal lens 180 that captures a hexagonal FOV 182, representing a pair of adjacent cameras such as may occur in a truncated icosahedron, or soccer ball-shaped panoramic multi-camera capture device (e.g., 100, 300), where the outer lens elements have pentagonal and hexagonal shapes. Logically, the hexagonal FOV 182 has a larger FOV near the apex but extends to a 20.9° half FOV, or 41.8° full FOV (□1), along the sides. The pentagonal FOV 177 supports a 36.55° FOV (□2) within a circular region, and the FOV is larger near the corners or apex. In particular, in this cross-section, the pentagonal FOV 177 is asymmetric, supporting a 20-degree FOV on one side of the optical axis 185 and only a 16.5-degree FOV on the other side of the optical axis.

[0030] Optical lenses are typically designed using programs such as ZEMAX or Code V. The success of the design typically depends in part on the selection of the best or most appropriate lens parameters, identified as operands for use in the merit function. This is also true when designing a lens system for an improved low-parallax multi-camera panorama capture device, for which there are several factors that affect performance (especially including parallax) and several parameters that can be optimized, either individually or as a whole, to control it. One approach aims to optimize the "NP" point, or more significantly, its variants.

[0031] As background, in the field of optical systems, there is the concept of the entrance pupil, which is the projected image of the aperture stop seen from the object space, or a virtual aperture through which the imaging rays from the object space appear to propagate towards it before any refraction by the first lens element. By standard techniques, the position of the entrance pupil can be found by identifying the paraxial chief ray from the object space 105 that passes through the center of the aperture stop and projecting or extending the object space in the direction in which it hits the optical axis 185. In an optical system, it is understood that the incident Gaussian or paraxial rays exist within an angular range of 10° or less from the optical axis, corresponding to the rays directed towards the center of the aperture stop, which also defines the entrance pupil position. Depending on the lens characteristics, the entrance pupil can be larger or smaller than the aperture stop and can be located in front of or behind the aperture stop.

[0032] In contrast, in the field of low-disparity cameras, there is the concept of a no-parallax (NP) point, or the center of perspective. Conceptually, the "NP point" is associated with the chief or principal ray that enters the outer edge or near the outer edge of the outermost lens element, and projects or extends its object space in the direction towards the position where it hits the optical axis 185. For example, depending on the design, the camera channels in a panoramic multi-camera capture device can support a half FOV with a non-paraxial chief ray at an angle of >31° for a dodecahedron type system (Figure 4), or >20° for a truncated icosahedron type system (see Figures 4 and 5A). This concept of NP point projection is applied to the design of panoramic multi-camera capture devices in relation to the expectations for chief ray propagation and parallax control for adjacent optical systems (cameras). It can also be said that when a camera rotates around the NP point, or when multiple cameras appear to rotate around a common NP point, the parallax error is reduced and the images can be aligned with little or no parallax error or perspective difference. However, in the field of low-disparity cameras, the NP point is also regarded as the same as the entrance pupil, and the axial position of the entrance pupil is estimated using the primary optical tangent relationship between the projection of the paraxial field of view angle at the first lens element (see Figures 2A and 2B) and the height of the incident light ray.

[0033] Thus, confusingly, in the field of low-disparity camera design, the NP point was previously associated with both the projection of the edge of the FOV chief ray and the projection of the chief ray within the Gauss or paraxial region. As you can see, in reality, both of them have value. Specifically, the NP point associated with the paraxial entrance pupil can be useful in developing initial specifications for designing and describing a lens. The NP point associated with the non-paraxial edge field chief ray can be useful in targeting and understanding parallax performance and in defining the conical volume or frustum within which the lens assembly can exist.

[0034] The projections of these non-paraxial chief rays can miss the entrance pupil defined by the paraxial chief rays due to both lens aberrations and practical geometric-related factors associated with these lens systems. In connection with the former, in a well-designed lens, the image quality at the imaging plane is typically prioritized by limiting the effects of aberrations on resolution, telecentricity, or other attributes. Within the lens system, aberrations at the intermediate surfaces, including the aperture stop, can vary widely as the net sum at the imaging plane is weighted. Aberrations at the aperture stop are often somewhat controlled to avoid vignetting, but non-paraxial chief rays do not need to pass through the center of the aperture stop or the entrance pupil placed in the projected paraxial direction.

[0035] To further explain these concepts and enable the design of an improved low-parallax lens system, note that the camera lens system 120 of FIG. 2A shows both a first NP point 190A corresponding to the entrance pupil defined by the vector projection of the paraxial chief ray from the object space 105 and an offset second NP point 190B corresponding to the vector projection of the non-paraxial chief ray from the object space. Both of these ray projections intersect the optical axis 185 at a position behind both the lens system and the imaging plane 150. As will be explained later, the ray behavior between and in the vicinity of the projection points 190A and 190B can be complex, and neither the projection position nor the projection points have definitive values and sizes. The projection of the chief ray intersects the optical axis at a point, but the projections of a group of chief rays converge towards the optical axis and intersect at different positions, which can be grouped tightly (e.g., within a few microns or tens of microns), in which case the extent or size of that "point" can be determined by the set of neighboring chief rays used in the analysis. On the other hand, when designing a low-parallax imaging lens that forms an image with a large FOV, the axial distance or difference between the NP points 190A and 190B provided by the projected paraxial and non-paraxial chief rays can be significantly large (e.g., millimeters). Thus, as will also be explained later, the axial difference represents a useful means for optimizing the parallax (e.g., low parallax amount 188) of lens systems designed for current panorama capture devices and applications. Similarly, as can be seen, the design of the improved device (300) can be optimized to position the geometric center of the device, or device center 196, outside, but close to, or alternatively, within, and preferably close to the non-paraxial chief ray NP point, of this low parallax amount 188.

[0036] In one aspect, FIG. 5A shows the projection of the theoretical edge of the field of view (FOV edge 155) past the outer lens elements (lenses 175 and 180) of two adjacent cameras to provide lines that converge to a common point (190). These lines represent the theoretical limits of a complex "conical" opto-mechanical lens assembly, which are typically pentagonal or hexagonal cones that limit the amount. Again, ideally, in a parallax-free multi-camera system, the entrance pupils or NP points of two adjacent cameras are located in the same place. However, to avoid mechanical collisions, the mechanism of a given lens assembly, including the sensor package, generally protrudes outside the frustum of the camera system and should not enter the conical space of an adjacent lens assembly. However, the actual lens assemblies within a multi-camera panoramic capture device are also separated by seams 160. Thus, the actual chief rays 170 that are received at the lens edges, inside both the physical width or aperture of the mechanical seam and the attached outer lens elements (lenses 175 and 180), when projected generally towards the paraxial NP point 190, instead arrive at an offset NP point 192 and can be separated by an NP point offset distance 194.

[0037] This can be better understood by considering an enlarged region A-A proximate to the nominal or ideal point NP190, as shown in detail in FIG. 5B. Within the hexagonal FOV182, a ray (e.g., paraxial ray 173) propagating within the Gaussian or paraxial region and passing through the nominal center of the aperture stop can be projected to the nominal NP point 190 (corresponding to the entrance pupil), or to an offset NP point 190A, with a small NP point difference or offset 193 from the nominal NP point 190. On the other hand, the actual hexagonal lens edge chief ray 170 associated with the largest inscribed circle within the hexagon can be projected to reach a common offset NP point 192A that can be at an even larger offset distance (194A). Two adjacent cameras of FIGS. 5A, 5B may or may not share the same NP point (e.g., 190) in the same location. The distance offset can result from various reasons, including geometric considerations between cameras (adjacent hexagonal and pentagonal cameras), geometric asymmetry within a camera (e.g., with respect to a pentagonal camera), or constraints from the practical width of the seam 160, or due to the directional differences between deviated rays.

[0038] As described above, there are also potential geometric differences in the projection towards the simplified nominal "NP point" (190) of the incident chief ray. First, the incident imaging optical path from near the corners or vertices or intermediate edges (mid-chords) of the hexagonal or pentagonal lens may or may not project to a common NP point within the described range between the nominal paraxial NP point 190 and the offset NP point 192B. Also, as shown in FIG. 5B, due solely to the geometric asymmetry of the pentagonal lens, the associated pair of chief rays 170 and 171 with respect to the actually accepted FOV can be projected to different nominal NP points 192B that can be separated from the paraxial NP point (190) by an offset distance 194B and from each other by an offset distance 194C.

[0039] As another problem, during lens design, the highest performance typically occurs on-axis or near the axis (e.g., ≤ 0.3 field of view (normalized)), near the optical axis 185. In many lenses, good imaging performance occurs, by design, often at or near the field edge, in which case optimization weighting is often used to enforce compliance. The worst imaging performance can, as a result, occur in the intermediate field (e.g., 0.7 to 0.8 of the height of the normalized imaging field). Considering FIGS. 5A and 5B again, although outside the paraxial region, intermediate off-axis rays from the intermediate field (θ), which are not as extreme as the marginal chief rays (10° < θ < 20.9°), can be projected towards an intermediate NP point between the nominal NP point 190 and the offset NP point 192B. However, other, more extreme, specifically, off-axis rays from the 0.7 to 0.8 intermediate field, which are more affected by aberrations, can be projected to an NP point at a position that is more or less offset from the nominal NP point 190 than the offset NP point 192B at the field edge. To absorb differences in lens design, the non-paraxial offset "NP" points can be placed either in front of (closer to the lens than) or behind (as shown in FIG. 2A) the paraxial NP point (entrance pupil) as proposed in FIG. 5B.

[0040] This is shown in more detail in FIG. 5C, which essentially illustrates a more zoomed-in region A-A of FIG. 5B, and illustrates the effects from the vector projection ray paths associated with deviated imaging rays that converge at or near the paraxial entrance pupil (190) for an imaging lens system designed and optimized using the method of the present approach. In FIG. 5C, the projected ray paths of the green deviated imaging rays in multiple fields of view from the camera lens system converge within a low disparity amount 188 near one or more “NP” points. A similar illustration of the ray fan can be generated for red or blue light. The projection of the paraxial ray 173 can converge at the nominal paraxial NP point 190, or on the nominal optical axis 185 at a distance Z behind the imaging plane 150, at or near the entrance pupil. The projection of the field edge ray 172, including the chief ray 171, converges at or near the offset NP point 192B along the optical axis 185. The NP point 192B can be defined quantitatively, for example, as the center of a large number of full field edge rays 172. An alternative offset NP point 192A corresponding to the “minimum confusion circle” can be identified, where the paraxial, edge, and intermediate or mid-field rays converge to a minimum spot. These different “NP” points are separated by offset distances 194A and 194B from the paraxial NP point, and by an offset distance 194C from each other. Thus, it can be understood that the aggregated “NP point” for any given actual imaging lens assembly or camera lens that supports a larger paraxial FOV, or an asymmetric FOV, is typically not a point, but instead can be an offset low disparity (LP) smudge or amount 188.

[0041] Within the smudge or low disparity amount 188, various conceivable optimal or preferred NP points can be identified. For example, the offset NP points corresponding to the peripheral field rays 172 can be emphasized to assist in providing improved image tiling. Alternative intermediate field (e.g., 0.6 - 0.8) NP points (not shown) can also be tracked and optimized. Also, the size and position of the entire "LP" smudge or amount 188, or the preferred NP points therein (e.g., 192B), can be changed according to the lens design optimization. Such parameters can vary between lenses for a given designed manufactured lens system due to manufacturing differences between lens assemblies. FIG. 5C shows that these alternative offset "NP points" 192A, B for non-paraxial rays are located behind the paraxial NP point 190 or further away from the lens and the imaging plane, but other lenses of this type optimized using the method of this approach can provide a similar non-paraxial NP point 192A arranged at the low disparity amount 188 in cases where it can occur between the imaging plane and the paraxial NP point.

[0042] FIG. 5C also shows the position with respect to the center of the low-disparity multi-camera panoramic capture device, device center 196. Based on optical considerations, the improved panoramic multi-camera capture device 300 can preferably be optimized to nominally position the device center 196 within the low-disparity amount 188. The optimized position therein may include being disposed either at or in proximity to either offset NP point 192A or 192B, or within the offset distance 194B therebetween, in order to prioritize parallax control with respect to the marginal field ray. The actual position is then determined by parallax optimization, which can be determined by lens optimization for spherical aberration of the entrance pupil, direct chief ray constraints, or distortion, or a combination thereof. For example, whether the spherical aberration is optimized to be over-corrected or under-corrected, and what weightings are used for the field operands in the merit function, can affect the positioning of the non-paraxial "NP" points for the peripheral or intermediate fields. The "NP" point positioning can also be determined by the management of manufacturing tolerances and residual differences in lens system manufacture. The device center 196 can also be disposed offset from the low-disparity amount 188 by the central offset distance 198, although in proximity to the low-disparity amount 188. This approach also aids in tolerance management and can provide additional space near the device center 196 for cables, circuitry, cooling hardware, and related structures. In such a case, the adjacent cameras 120 can, as a result, have an offset low-disparity amount 188 of the "NP" points (FIG. 5D) instead of being in the same location (FIGS. 5A, 5B). In this example, if the device center 196 is instead disposed at or in proximity to the paraxial entrance pupil, NP point 190, one or more of the outer lens elements 137 of the camera 120 are substantially smaller than normal and the desired full FOV is not achievable. FIG. 13B shows possible positionings of a similar lens system 920 with respect to the offset device center 910.

[0043] Therefore, the zero-parallax (NP) point is a useful concept to work with, providing useful information for panoramic image capture and system design, and can assist in the design of low-parallax error lenses, which is ideal and the constraints thereof need to be understood. Considering this explanation of the NP point(s) and the LP smear, when enabling an improved low-parallax multi-camera panoramic capture device (example of lens design to follow; device 300 of FIG. 13A), it is important to understand the ray behavior in this region and define the appropriate parameters or operands to be optimized, as well as the appropriate target levels of performance to aim for. In the latter case, for example, a low-parallax lens with a track length of 65 - 70 mm can be designed for a case where the LP smear is 10 mm wide (e.g., offset distance 194A). However, alternative lens designs where this parameter is further improved may have a low-parallax amount 188 with a longitudinal LP smear width or a width along the optical axis (offset 194A) of less than a few millimeters.

[0044] The width and position of the low disparity amount 188, as well as the vector directions of the projections of the various chief rays, and their NP point positions within the low disparity amount, can be controlled during lens optimization by a method using the operands associated with the chief ray fan (e.g., FIGS. 2A, 2B). However, the LP smear or LP amount 188 of FIG. 5C can also be understood as the visualization of the lateral component of the spherical aberration of the entrance pupil, and this parameter is an alternative but equivalent design optimization method that can be used instead of using the chief ray fan. Specifically, during lens optimization, for example, using Code V, the lens designer can create a special user-defined function or operand for the lateral component of the spherical aberration of the entrance pupil (e.g., the height of the ray), which can then be used in various ways. For example, the operand value can be calculated as the residual sum of squares (RSS) of the values across the entire FOV or a local field of view, using either uniform or non-uniform weighting for the field of view operand. In the latter case of local field of view preference, the values can be calculated for a position at or near the entrance pupil or anywhere within the low disparity amount 188, depending on the preference for the paraxial, intermediate, or peripheral fields of view. An equivalent operand can be the width of the minimum confusion circle in a plane, such as the plane of the offset NP point 192A or the plane of the offset NP 192B as shown in FIG. 5C. The optimization operand can also be calculated with a weighting to reduce or limit the disparity error non-uniformly across the field of view, with an unbalanced weighting that favors the peripheral or marginal fields of view over the intermediate field of view. Alternatively, the optimization operand can be calculated using weighting to provide a nominally low disparity error in a nominally uniform manner across the entire field of view (within or across the core FOV 205 as in FIG. 7). That type of optimization can be particularly useful for mapping applications.

[0045] Regardless of whether the low-parallax lens design and optimization method uses an operand based on the spherical aberration of the chief ray or the entrance pupil, the resulting data can also be analyzed in relation to changes in the imaging perspective. Specifically, parallax errors for the field of view and color can also be analyzed using the calculation of the center of perspective (COP), which is a parameter more directly related to visible image artifacts with a low amount of parallax and can be evaluated by the image pixel error or difference for imaging an object at two different distances from the camera system. The center-of-perspective error is essentially a change in the chief ray trajectory given a plurality of object distances - an object at a near distance (3 ft), another object at "infinity", etc.

[0046] In figures and architectures, perspective is a technique for depicting solid objects on a two-dimensional surface in order to give the correct impression of their height, width, depth, and position in relation to each other as seen from a particular point. For example, in a figure using linear or point perspective, objects appear smaller as their distance from the observer increases. Such illustrated objects are also subject to foreshortening perspective, meaning that the dimension of the object along the line of sight appears shorter than its dimension across the line of sight. Perspective functions by representing the light from the scene to the viewer's eye as if the viewer were looking through a virtual rectangle (realized as the plane of the figure) and directly drawing what is seen onto the window glass.

[0047] Perspective is related to both parallax and stereopsis. In stereoscopic image capture or projection using a pair of adjacent optical systems, perspective is a visual cue, along with dual-view parallax, shading, and occlusion, that can provide a sense of depth. As already described, parallax is the visual perception that the position or orientation of an object appears different when viewed from different positions. In the case of image capture by a pair of adjacent cameras with at least partially overlapping fields of view, the parallax image difference is a cue for stereoscopic image recognition or an error for panoramic imaging assemblies.

[0048] Whether it is a camera or the human eye, in order to capture an image using an optical system, the geometry and performance of the optical system affect the usefulness of the resulting image with respect to low parallax (panorama) or high parallax (stereo) recognition. Specifically, for an ideal lens, all chief rays from the object space point exactly towards the center of the entrance pupil, and the entrance pupil coincides with the center of perspective (COP) or the center of the viewpoint with respect to the resulting image. There is no error in perspective or parallax with respect to such an ideal lens.

[0049] However, for an actual lens, since it has both physical and image quality constraints, residual parallax errors can exist. As described above, for an actual lens, the projection of the paraxial chief ray from the first lens element points towards a common point, the entrance pupil, and its position can be determined as the axial distance from the front surface of that first element. On the other hand, for an actual lens that captures a sufficiently large FOV to include non-paraxial chief rays, the chief rays in the object space can point towards a common position or a nearby quantity, but typically point towards a position offset from the center of the entrance pupil. These chief rays do not essentially coincide at a single point, but they can be directed through a small low-parallax amount 188 (e.g., LP "smudge") by appropriate lens optimization. The vertical or axial variation of the rays within the LP smudge can be determined from the position of the chief ray across the optical axis. The ray error can also be measured as the lateral width or axial position of the chief ray within the LP smudge.

[0050] The concept of parallax correction is illustrated in FIG. 5D with respect to the center of perspective. The first camera lens 120A collects light from the object space 105, including light from two outer light ray fans 179A and 179B whose chief ray projections converge towards a low parallax amount 188A, and forms an image in at least one core FOV. These light ray fans may correspond to near the field of view edge or a group 172 of field of view edge rays, as shown in FIG. 2B or FIG. 5C. As shown in FIG. 5C, within the LP amount 188, the vector projection of such rays from the object space, generally directed towards the image space, favors the field of view edge rays, so that it can cross the optical axis 185 beyond the imaging plane at or near an alternative NP point 192B that is selected or can be preferred. However, as also shown in FIG. 5C, such field of view edge rays 172 do not need to intersect the optical axis 185 at exactly the same point. These differences, when transformed back to the object space 105, are converted into small differences in parallax or perspective within or across the imaging ray bundle or fan for the imaging FOV (e.g., core FOV 205 as in FIG. 7) of the camera lens.

[0051] The second adjacent camera lens 120B, shown in FIG. 5D, provides similar performance and can image a fan of chief rays 170 from within the core FOV 205 that have vector projections of these chief rays converging within a corresponding low parallax amount 188B, including a light ray fan 179C. The LP amounts 188A and 188B can overlap, or coincide, or be offset depending on factors including the camera geometry and the seam between adjacent cameras, or lens system manufacturing tolerances and compensators, or whether the device center 196 is offset from the LP amount 188. The more these LP amounts 188 overlap or coincide, the more the centers of perspective of the two lens systems overlap. The light ray fan 179B of the camera lens 120A and the light ray fan 179C of the camera lens 120B are nominally parallel to each other, e.g., there is no parallax error between them. However, even if only a very small residual parallax error at the FOV edge is recognized in the lens design, that difference can be increased due to manufacturing differences between the lens systems.

[0052] Analytically, the chief ray data from the actual lens can also be expressed as a function of the field of view in terms of perspective error, including color error. Perspective error can thus be analyzed as a positional error in the image between two objects placed at different distances or in different directions. Perspective error can be determined by the choice of COP position, the angle within the imaging FOV, and color error. For example, it may be useful to prioritize the COP in order to minimize the green perspective error. Perspective error or parallax error can be reduced by optimizing the axial position (Dz) or width of the color within the LP amount 188 associated with the center of perspective for one or more fields of view within the imaging FOV. The center of perspective can also be graphed and analyzed as a family of curves for each color of the Z (axial) intercept position (distance in mm) versus the field of view. Alternatively, to get a better idea of how the captured image will look, the COP can be graphed and analyzed as a family of curves for the camera system as parallax error at the image pixels for each color, contrasted with the field of view.

[0053] During the design of the camera lens system, the goal can be to limit the parallax error to below a few pixels for the imaging within the core FOV 205 (Figure 7). Alternatively, for example, it can be preferable to limit the parallax error, especially in the peripheral field of view, with respect to the outer edge of the core FOV and with respect to the extended FOV region (if provided). If the residual parallax error for the camera is accordingly small enough, the parallax difference seen as a perspective error between two adjacent cameras, near their shared seam 160 or within the seam-related region of the extended FOV where the imaging overlaps, can similarly be limited to below a few pixels (e.g., ≤ 3 - 4 pixels). Depending on the lens design, device design, and application, it can be possible and preferable to further reduce the parallax error for the lens system to ≤ 0.5 pixels for the entire core FOV, the peripheral field of view, or both, as measured by the perspective error. If these residual parallax errors for each of two adjacent cameras are small enough, the images can be acquired, trimmed, and easily tiled while correcting or hiding image artifacts from any residual seam 160 or blind region 165.

[0054] Continuing with the design of that type of panoramic camera of FIG. 1, the selection of lens optimization methods and parameters can be important in order to enable an improved low-parallax multi-camera panoramic capture device (300) having a plurality of adjacent cameras. A system of camera lenses 120, or lens elements 135, such as that of FIG. 2A, can be used as a starting point. The camera lens has a compressor lens element(s), and an inner lens element 140, which can also be defined as being composed of a wide-angle lens group in front of the aperture, and an eyepiece-like lens group behind the aperture. In the design of such lenses to reduce parallax error, it can be useful to consider how the fan of chief rays 125 (see FIG. 2A), or the fan of marginal chief rays 170 (see FIG. 2B), or the local collection 172 (see FIG. 5C) or 179A, B (see FIG. 5D) of field-edge rays is imaged by the camera lens assembly. It is possible to optimize the lens design by using a set of merit function operands for a set of chief rays (e.g., 31 defined rays), but the optimization process can, as a result, become cumbersome. Alternatively, in the pursuit of the design of an improved low-parallax multi-camera panoramic capture device (300), improved performance has also been confirmed to be obtained by using a reduced set of ray parameters or operands that emphasize the lateral component of spherical aberration at the entrance pupil or at a similar selected surface or position (e.g., offset NP points 192A or 192B) within the LP smear amount 188 behind the lens system. Optimization for the lateral component of spherical aberration at an alternative off-axis entrance pupil can be achieved using merit function weighting that emphasizes off-axis chief rays.

[0055] As another aspect, in the low-parallax multi-camera panoramic capture device, the fan 170 of the chief rays (see FIG. 2B) incident on or near the beveled edge of the outer lens element of camera 120 should be parallel to the fan 170 of the chief rays (see FIG. 1) incident on or near the edge 132 of the beveled surface of the outer lens element of the adjacent camera. The "edge" of the outer lens element 137 or the compressor lens is a three-dimensional structure (see FIG. 2B), which may have a flat edge across the thickness of the glass, and it should be noted that it is subject to the manufacturing tolerances of the lens element, the entire lens assembly, and the housing 130, as well as the adjacent seam 160 and its structure. The definition of the position where the beveled edge is cut into the outer lens element is determined by factors including material properties, front color, distortion, parallax correction, tolerances, and the range of any additional extended FOV 215. The outer lens element 137 becomes a faceted outer lens element when the beveled edge 132 is cut into the lens, and nominally creates a set of polygonal edges following a polygonal pattern (e.g., pentagonal or hexagonal).

[0056] As a result, the camera system 120 having an outer lens element with a polygon for capturing incident light from a polygonal field of view forms a polygonal image on the imaging surface 150, and the shape of the captured polygonal field of view nominally matches the shape of the polygonal outer lens element. The cuts of these beveled edges for a given pair of adjacent cameras can affect both the imaging and the opto-mechanical structure at or near the intervening seam 160.

[0057] As another aspect, FIG. 5E shows the “front color,” which is the difference in the nominal ray path by color versus field of view directed to an off-axis or marginal field point. Typically, for a given field point, the blue ray is offset the farthest. As shown in FIG. 5E, the blue ray 157 received on the first lens element 137 is at a DX about 1 mm further out than the received red ray 158 directed to the same imaging field point. If the lens element 137 is not large enough, this blue light can be clipped or blurred, and color non-uniformity artifacts can occur at or near the edge of the imaging field. The front color can appear within the captured image content as the contour of a polygon FOV or a polygon edge rainbow of the outer compressor lens element 437 that functions as a field stop for the optical system. The local color transfer difference that can cause front color-related color non-uniformity artifacts near the image edge can occur due to differential vignetting at the sloped edge of the outer compressor lens element 137, or from edge truncation in the compressor lens element 438 (FIG. 13A), or through the aperture stop 145. During lens design optimization to provide an improved camera lens (320), the front color can be reduced (e.g., to a DX ≤ 0.5 mm width) as part of the lens design's color correction, including by glass selection within the compressor lens group or the overall lens design, or as a trade-off in the correction of lateral color. The effect of front color on the captured image can also be reduced opto-mechanically by designing the improved camera lens (320) to have an extended FOV 215 (FIG. 7) and the opto-mechanics to push a straight or sloped lens edge 132 at or beyond the edge of the extended FOV 215 so that residual front color occurs outside the core FOV 220. As a result, the front color artifacts can be removed during the image trimming step in image processing. The effect of front color or lateral color can also be reduced by spatially varying color correction during image processing.As another option, an improved camera lens (320) can have a color-dependent aperture that provides a larger transmission aperture (diameter) for blue light than for, for example, red or green light, at or near the aperture stop.

[0058] The optical performance at or near the seam can be somewhat understood in relation to distortion (Figure 6) and a defined set of fields of view (Figure 7). Specifically, Figure 7 shows a potential set of fields of view that potential image light can be collected by two adjacent cameras. As an example, a camera with a pentagonal outer lens element having a seam 160 that separates it from an adjacent lens or camera channel, whether associated with an icosahedron or truncated icosahedron or other polygonal lens camera assembly, can image an ideal FOV 200 that extends to the apex (60) or polygonal edge of the frustum or conical volume in which the lens is present. However, due to various physical constraints that can occur at the seam, including the finite thickness of the lens housing, the physical aspects of the sloped lens element edge, mechanical wedges, and tolerances, a smaller core FOV 205 of the passing image light can actually be imaged. The coated aperture for the outer lens element 137 should surround at least the core FOV 205 with some margin (e.g., 0.5 - 1.0 mm). Since the lens can be manufactured with an AR coating before beveling, the coating can extend to the seam. The core FOV 205 can be defined as the largest low-parallax field of view that a given actual camera 120 can image. Equally, the core FOV 205 can be defined as a sub-FOV of a camera channel whose boundaries are nominally parallel to the boundaries of its polygonal cone (see Figures 5A and 5B). Ideally, with a small seam 160, as well as proper control and calibration of the FOV pointing, the nominal core FOV 205 approaches or matches the ideal FOV 200 in size.

[0059] During the camera alignment and calibration process, a series of image references 210 can be established along one or more of the edges of the core FOV 205 to assist with image processing and image tiling or mosaicking. The resulting gap between the core FOV 205 supported by the first camera and that supported by an adjacent camera can result in a blind region 165 (Figs. 5A, 5B). To correct for the blind region 165 and the associated loss of image content from the scene, the camera can be designed to support an extended FOV 215, which provides sufficient extra FOV to absorb the seam width and tolerances, or the offset device center 196. As shown in Fig. 7, the extended FOV 215 can be extended sufficiently to provide an overlap 127 with the edges of the core FOVs 205 of adjacent cameras, although the extended FOV 215 can be made even larger. This limited image overlap can result in a small amount of loss of image resolution, parallax error, and some complication in image processing, as described above with respect to Fig. 3, but can also help to reduce the apparent width of the seam and the blind region. However, as provided by this approach, if the extra overlap FOV is modest (e.g., ≤ 5%) and the residual parallax error therein is sufficiently small (e.g., ≤ 0.75 pixel perspective error), the image processing burden can be very small. Image capture up to the extended FOV 215 can also be used to enable an intermediate capture step that supports camera calibration and image correction during operation of the improved panoramic multi-camera capture device 300. Fig. 7 shows an inscribed circle within one of the FOV sets that corresponds to a subset of the core FOV 205, which is a common core FOV 220 that can be captured in all directions from that camera. The angular width of the common core FOV 220 can be useful as a quick reference to the image capacity of the camera. An alternative definition of a larger common core FOV 220 may also be useful to include the entire core FOV 205.From the common core FOV 220 or the core FOV 205, beyond the ideal FOV 200, the dashed line (225) that extends to nominally include the extended FOV 215 represents a region that can support careful mapping of the chief or principal rays or control of the spherical aberration of the entrance pupil in the lens design to enable low parallax error imaging and easy tiling of the images captured by adjacent cameras.

[0060] Over the seam 160 that extends across the distance between two adjacent usable apertures between two adjacent cameras, it can be advantageous if the image light is captured substantially straight, parallel, and at a common interval with respect to the finite distance to reduce parallax and improve image tiling. The amount of FOV overlap required to provide an extended FOV to limit blind regions can be determined by controlling the relative proximity of the entrance pupil (paraxial NP point) or an alternate preferred plane (e.g., to emphasize peripheral rays) within the low parallax amount 188 with respect to the device center 196 (e.g., the center of the dodecahedron shape). The amount of the extended FOV 215 is preferably 5% or less (e.g., an additional field of view of 1.8° or more for a nominal core FOV of 37.5°), whereby the peripheral field of view of the camera is thus, for example, about 0.85 to 1.05. If the spacing constraints at the device center and the manufacturing tolerances are well managed, the extended FOV 215 can be reduced to an additional field of view of 1% or less. Within the extended FOV 215, the parallax should nominally be limited to the system level, while on the other hand, both the image resolution and the relative illuminance remain satisfactory. Parallax optimization for reducing parallax error can use either the chief ray or the pupil aberration constraint, targeting the optimization for the high FOV region (e.g., 0.85 to 1.0 field of view) or including the extra camera overlap region provided by the extended FOV 215 (e.g., in FIG. 7, the partial field of view range of about 0.85 to 1.05) beyond that.

[0061] In addition, when enabling an improved low-parallax multi-camera panoramic capture device (300) with limited parallax error and improved image tiling, it may be important to control image distortion for image light passing at the edge of the FOV of the outer lens element, e.g., in the peripheral field of view or near it. In geometric optics, distortion is a deviation from a preferred state where straight lines in the scene remain straight in the image (e.g., linear projection). It is a form of optical aberration that describes how light rays from the scene are mapped onto the imaging plane. Generally, for image capture, in lens assemblies used for human viewing, it is convenient to limit image distortion to a maximum of + / - 2%. In current applications, having a moderate distortion of 2% or less may also be useful for tiling or combining panoramic images from images captured by adjacent cameras. By reference, in barrel distortion, the image magnification decreases with the distance from the optical axis, and the obvious effect is that of the image mapped around a sphere (or cylinder). Fisheye lenses, often used to capture hemispherical or panoramic views, typically have this type of distortion as a way of mapping an infinitely wide object plane onto a finite image area. Fisheye lens distortion (251) can be significant as a deviation from f-θ distortion (e.g., 15% or 90° half-width (HW) over the full field of view), but is only a few percent for small fields of view (e.g., ≦ 30° HW). As another example, in laser printing or scanning systems, f-θ imaging lenses are often used to print images with minimal banding artifacts and image processing corrections for pixel placement. Specifically, F-θ lenses are designed using barrel distortion that results in a linear pixel positioning with a substantially constant spot or pixel size and a field of view angle θ (h = f * θ).

[0062] Accordingly, an improved low-parallax camera 320 that captures a half FOV of 35 to 40° or less may have a fisheye distortion 251, as the distortion may be sufficiently low. However, the distortion can be more favorably optimized for the design of an improved camera lens assembly for use in an improved low-parallax multi-camera panoramic capture device (300). As a first example, as shown in FIG. 6, it may be advantageous to provide a camera lens assembly having a local nominal f-θ distortion 250A at or near the edge of the imaging field. In one example, the image distortion 250 peaks at about 1% in a field of view of about 0.75, and the lens design is not optimized to provide an f-θ distortion 250 below about 0.85 field of view. However, during the lens design process, the merit function can be constrained to provide a distortion 250A such as a nominal f-θ or a substantially flat distortion 250B for light rays imaged at or near the field of view edge, such as for the peripheral field of view that extends over a partial field of view range of about 0.9 to 1.0. This range of high fields of view with f-θ type or flat distortion correction includes the fan 170 of chief rays or peripheral rays of FIG. 2B, including light rays imaged through the corners or vertices 60, such as those of a lens assembly with hexagonal or pentagonal outer lens elements 137. Additionally, for manufacturing tolerances and dynamic effects (e.g., temperature changes) applicable to both the camera 120, which includes both the lens element 135 and the housing 130, and the collection of cameras 120 within the panoramic multi-camera capture device, it may be advantageous to extend the region of nominal f-θ or flattened distortion in the peripheral field of view beyond the nominal full field of view (e.g., 0.85 to 1.05). This is shown in FIG. 6, in which the region of reduced or flattened distortion extends beyond the full field of view to about 1.05 fields of view. In such a peripheral field of view range, it may be advantageous to limit the total distortion change to 0.5% or less. Control the peripheral field of view distortion to keep the image "edge" straight within an adjacent pentagonal-shaped region. This may enable more efficient use of pixels and thus faster image processing when tiling the images.

[0063] The foregoing description discusses distortion in the classical sense as image aberration in the imaging plane. However, in a low-parallax camera, this residual distortion is typically a trade-off or nominal cancellation of the contribution from the compressor lens elements (137, or 437 and 438 in FIG. 13A) relative to that of the collective inner lens element (140, or 440 in FIG. 13A). Importantly, the ray direction change caused by the distortion contribution of the outer compressor lens element also affects both the imaging ray path and the projection chief ray path towards the low-parallax amount. As a result, for at least some low-parallax lens designs, this means that distortion optimization can affect parallax or field-of-view edge NP point or perspective center optimization.

[0064] To assist with image tiling, the definition of the peripheral or partial field-of-view range (e.g., including an additional field-of-view of about 0.85 - 1.05, or 5% or less) where parallax, distortion, relative illuminance, resolution, and other performance factors are carefully optimized can depend on the device and camera geometry. As an example, for a hexagonal lens and field-of-view, the lower limit of the peripheral field-of-view can be defined as about 0.83, and for a pentagonal lens as about 0.8. FIG. 7 illustrated an example with two adjacent pentagonal outer lens elements and FOV sets, but the approach of defining the peripheral field-of-view and extended FOV to support a small area of overlapping image capture is applicable to multi-camera capture device designs with adjacent pentagonal and hexagonal cameras, or adjacent pentagonal cameras, or other cameras with adjacent edges of other polygonal or arbitrary shapes or contours in general.

[0065] To make the extended FOV 215 functionally useful, the nominal image formed on the image sensor corresponding to the core FOV 205 needs to be underfilled by at least enough to enable the extended FOV 215 to also be imaged on the used image area of the image sensor. This can be done to accommodate actual differences from the ideal of the manufactured lens assembly, or design with the offset device center 196, and manufacturing differences in the improved low-parallax multi-camera panoramic capture device (300). However, as will be explained later, a clever mechanical design of the lens assembly can affect both the imaging field of a given camera and the seam between cameras, limiting mechanical displacement or wedging, and helping to reduce parallax errors and FOV overlap or underlap. Similarly, compensation or adjustment at the reference of the image FOV (core FOV 205) size and position, as well as image centroid tracking and shape tracking, can be useful. Combining in some combination the optimization of distortion for the extended peripheral field of view and low or zero parallax imaging, careful mechanical design to limit and correct component and assembly differences, and the use of correction references or compensators can provide an excellent overall system solution. As a result, the images captured from the camera can be easily trimmed to the nominal size and shape expected for the nominal core FOV 205, and the images from multiple cameras can then form a mosaicked or tiled panoramic image together with a reduced burden on image post-processing. However, the extended FOV 215 should, if necessary, provide sufficient additional angular width (e.g., q1 ≦ 5% of the FOV) to match or exceed the expected wedge or tilt angle q2 that can occur at the seam (q1 ≧ q2).

[0066] In the design of an improved imaging lens of a type that can be used in a low-parallax panoramic multi-camera capture device (100 or 300), several primary parameters can be calculated to inform the design effort. The main parameters are the target size of a frustum or cone volume based on the selected polygonal configuration (lens size (FOV) and lens shape (e.g., pentagonal)) and the sensor package size. Other main parameters that can be estimated include the nominal position of the paraxial entrance pupil, the focal lengths of the compressor lens group and the wide-angle lens group, and the FOV seen by the wide-angle group.

[0067] However, the design optimization for an improved camera lens (320) for use in an improved low-parallax panoramic multi-camera capture device (300) also depends on how a number of other lens attributes and performance metrics are prioritized. Specifically, the relevant system parameters are optimized using the chief ray fan or spherical aberration of the entrance pupil and may include control of parallax or center of perspective (COP) errors at the edge of the imaging field or inner field positions or both. These parameters are closely related to other important parameters including the width and position of the "LP smear" or quantity 188, the size of any central offset distance between the entrance pupil or LP smear and the device center 196, the target width of the gap or seam, the extent of the blind region 165, and the size of any peripheral or extended FOV to provide overlap. The relevant performance metrics may include image resolution or MTF, distortion (particularly in the peripheral field of view and the distortion of the first compressor lens element and compressor lens group), lateral color, relative illumination, front color, and color vignetting, telecentricity, and ghosting. Other relevant design variables may include mechanical and material parameters such as the number of compressor lens elements, the configuration of the compressor lens group, wide-angle lens group, and fish-eye lens group, glass selection, the maximum allowable size of the first compressor or outer lens element, sensor package size, track length, the nominal distance (e.g., working distance) from the imaging plane to the nearest preceding lens element, the nominal distance from the imaging plane to the entrance pupil, the nominal distance from the imaging plane or entrance pupil to the polygon center or device center, manufacturing tolerances and limitations, and the use of compensators.

[0068] Figure 13A provides a cross-sectional view of an alternative and improved opto-mechanical design for an improved camera 320 that can be used in the improved panoramic multi-camera capture device 300. In this exemplary design, the camera lens 320 has a lens form in which the compressor lens elements are divided into a compressor lens group including first and second compressor lens elements (437 and 438). The inner lens element 400 includes a wide-angle lens group disposed in front of the aperture stop 445, which includes a fourth lens element 442 and an eyepiece lens group behind the stop. As described above, the lens system provides a paraxial NP point 490 that is offset from the non-paraxial chief ray NP point 492 located within the low parallax amount 488. The size or width of this amount, and the position of the NP points of potential interest therein (e.g., paraxial, intermediate field of view, peripheral field of view, minimum confusion circle based), are determined by design priorities and parallax optimization (e.g., spherical aberration of the entrance pupil, chief ray fan, distortion). The lens forms of the camera lens system design examples of FIGS. 2A and 13A are similar, but the lenses vary in detail and performance, including with their different compressor lens configurations. Due to differences in specifications, optimization methods, and priorities, these lenses also differ in cost, performance, and manufacturability.

[0069] Depending on the priority, these lens systems can be further optimized, and the various differences in lens form may be individually better suited for different markets or applications. Generally, the outermost lens element, or the first compressor lens element, uses Ohara SLAH52, SLAH53, and SLAH63 glass (or equivalent glass from Schott Glass (e.g., N-LAF36 and N-LASF43)), which are high-index, low-dispersion flint glasses with a refractive index n of about 1.80 and an Abbe number Vd of about 41.5 in the visible spectrum. It should be understood that other optical materials can be used for the lens elements generally in the camera lens 520 including the compressor lens element. For example, the use of high-index, low-dispersion, mid-crown glass such as Ohara SLAL-18 may be useful for color correction. As another example, the lens elements can also be made from optical ceramics such as Alon (n about 1.79, Vd about 57-72) or spinel, which are extremely durable materials similar to sapphire but have excellent light transmittance, low dispersion, and a controllable and modifiable isotropic crystal structure. It should also be understood that the camera lenses of this approach can be designed with refractive, gradient, glass or optical polymer, reflective, aspherical or freeform, kinoform, Fresnel, diffractive or holographic, sub-wavelength or metasurface, optical elements that consist of or include optical properties. These lens systems can also be designed using achromatic or apochromatic color correction, or thermal defocus desensitization.

[0070] FIG. 13A does not show how an improved camera lens can be mounted within the lens housing, but the proximity of the large outer compressor lens element 437 and the large second compressor lens element 438 forming a doublet may require attention to securely support these closely spaced elements. As one approach, the first compressor lens element 437 may be positioned centrally with respect to the compressor doublet (438), and the doublet may be centered on a major circular datum on the inner surface of the lens housing. This datum may be a tight tolerance with respect to the channel centering hub 330 to reduce tolerance accumulation between adjacent camera channels.

[0071] Improving the optical design of a camera lens system is important to enable an improved low-parallax panoramic multi-camera capture device (300), but improving the optomechanical design can also be equally important. As noted in the foregoing proposals, the actual performance of camera 120 can differ from the designed performance due to material and manufacturing differences between the individual lens elements 135 and housing 130 and their components, and their interactions. As a result of such differences, image quality (e.g., aberrations, including distortion), focal length (EFL) and magnification, working distance or track length, beam aiming or imaging position, and other attributes of camera 120 can vary. These differences can also mean that the assembly and performance of a given camera can be different from that of another camera with a nominally identical optomechanical design. For example, the focal length of a set of nominally identical cameras can vary by up to ±2%, which in turn causes similar variations in lens magnification and FOV. This variation can be reduced or eliminated by designing the improved camera lens to be a variable focus type, including a focal length compensator, such as by using a lens element whose axial position can be adjusted. Alternatively, camera 120 can be designed such that the nominal image from the nominal camera underfills the image sensor, so that an image from a camera with a large (e.g., +2%) focal length lens also sufficiently underfills the sensor, despite having little margin. During calibration to determine the FOV, the EFL or magnification of the lens can be measured and the sensor can also be aligned to be in focus with that lens. Image processing software can then be used to correct the image for lens differences, including correction of the image size for magnification and distortion variations between lenses.

[0072] Considering optomechanics in more detail, the axial alignment or focus position of the image sensor relative to the imaging plane 150 provided by the camera lens assembly (320) can be improved by an appropriate mechanism. For example, FIGS. 8 and 9 show a portion of an improved panoramic multi-camera capture device 300, in which the image sensor 270 can be assembled into a sensor package 265, and the sensor package 265 includes a plate 290, a circumferential flange, several adjustment screws 280, a flexure, or springs 285. For example, three adjustment screws can be used to control the X translation in conjunction with the Z-axis rotation, another set of three screws can be used to control the Z-axis translation in conjunction with the X-Y axis rotation, and additional screws are used to control the Y-axis translation. A pair of springs is used to hold the gimbal plate 290 and at the same time allow X and Z-axis translations respectively. Other adjustment designs or devices can be used within the narrow space constraints allowed by the entire camera 320 and panoramic multi-camera capture device 300, such as using pins and micrometers or pins and shims.

[0073] Manufacturing differences for individual cameras, and opto-mechanical interactions between them, can have a significant impact on the design and performance of the multi-camera capture device 300. During the initial assembly of the multi-camera capture device 300, tolerances and mechanical wedge interactions between the housing 430 of the first camera 320 and the housing 430 of the second adjacent camera 320 can affect the seam 400, the alignment of the core FOV 205 or the extended FOV 215 of the individual cameras (causing an overlap or underlap of the FOVs), and thus affect the FOVs captured by each camera. Further, the mutual alignment mounting stresses applied to adjacent cameras by the close mounting physically deform one or more of the camera housings, which can then potentially distort or skew the optical imaging function of the camera lens as well. As a result, the camera lens system can provide an image to a sensor that is shifted, rotated, or tilted out of plane. These potential problems, or the risk of their occurrence, can be exacerbated by environmental effects such as asymmetric thermal loads or substantial asymmetric optical loads.

[0074] To counter such problems, an improved multi-camera capture device 300, as shown in FIG. 9, may include features for providing kinematic mounting of individual cameras 320 or objective lenses. Specifically, FIG. 9 shows two views of an icosahedral multi-camera capture device 300, including a partial cross-sectional view in which eleven pentagonal cameras 320 are attached to a central support 325 that occupies the nominal position of a twelfth potential camera channel. Each camera 320 has a separate base lens assembly or housing 430 consisting of a lens mount that attaches a compressor lens (437) and also attaches an inner lens element 440 that together form the base lens assembly. For each camera 320, the lens elements and housing 430 fit within a nominal conical space or volume, although they need not nominally fill that space. In practice, the sharp ray bending introduced by the compressor lens element underfills the available space with the inner lens elements 440 and their housings or barrels, and the entire lens housing 430 may further taper inwardly, potentially leaving an open internal volume 390 between adjacent lens assemblies.

[0075] The housing 430 or base lens assembly of FIG. 9 also includes a turned section that can be machined on a CNC multi-axis (5-axis) machine and engages a channel centering hub 330 like a tripod. The channel centering hub 330 can be fully machined on a lathe except for the pentagonal flange, which is completed in a finishing operation after the lathe work. Machining on a lathe means that exceptional concentricity and runout can be achieved, which helps with the ultimate alignment of the channels. The housing 430 engages the inner diameter of the channel centering hub 330, which is a major part of a central mounting mechanical assembly designed to mate with it, ranging from a slip fit to a light interference fit, to ensure axial alignment without significant variation due to clearance tolerances. This same fit reduces the perpendicular error with respect to the channel axis.

[0076] The tripod or channel centering hub 330 also includes a swivel or ball pivot that engages the socket 345 of the spherical socket array 346 provided on the central support 325. In this system, the cameras 320 located at the extreme positions, on the opposite side of the central support 325, are precisely positioned reference channels. The central support 325 consists of a cylindrical post with a ball on top. The geometry of the central support 325 and the tripod or centering hub 330 can be designed to provide more space for power and communication cables, cooling pipes, and a mechanism for fixing the lens housing 430 or cables. In this example, the balls include sockets 345, each of which can receive a ball pivot 340. The ball pivot 340 is at the end of an extension pin or ball pivot arm 342. This ball and socket portion on the central support 325 base can be costly to machine, but given the expected accuracy regarding the position and depth of the sockets, the advantage is that the centerline pointing is controlled, while there is only one part per device 300 that requires special accuracy. On the other hand, each of the camera channels 320 can be machined with low accuracy, which reduces both manufacturing and replacement costs.

[0077] Each of the individual camera lens housings 430 of FIG. 9 is provided with an external or outer inter-channel datum 335 disposed mid-way along the pentagonal side. These inter-channel datums 335 can include two parallel, slightly curved, protruding bars that are separated by a groove therebetween. Both of these datums are designed to provide single-point or local kinematic contact or interaction between the lens housings, such that the datum features are interlocked in a manner such that only one part or housing is advantaged from a tolerance perspective. Since they are interlocked, only the variation of one part affects the distance between each camera channel, and thus, the angle between the channels. Specifically, if one of the datums 335 is large, the other does not contact, and it is advantaged. Thus, only one tolerance contributes to the two parts. The inter-channel datums 335 being interlocked from one camera 320 to another also limits the lateral movement between the engaged (pentagonal) faces or sides while allowing for limited angular movement of the lens housing 430.

[0078] Individually and collectively, the interaction between the camera lens housings 430 or the base lens assemblies limits mechanical displacement and wedge or channel pointing errors (roll, pitch, and yaw) between the cameras due to both the ball and socket arrangement and the datum features (335). Each camera channel assembly functions with its adjacent assembly to limit channel pointing errors. The portion of the base lens assembly (430) that holds the outer lens element 437 or the compressor lens also has an inner functional datum that can position the compressor lens perpendicular to the optical or mechanical channel axis, and it has additional inner datum features that limit axial displacement. As will be described in detail later with respect to FIG. 16, an additional set of alignment features provided on the polished edge of the compressor lens functions as a datum and can interact with these inner datum features to also limit rotation about the pentagonal channel axis.

[0079] The use of the alignment features shown in FIG. 9, and specifically the ball pivot and socket datums (350 and 356) and the channel-to-channel datum feature 335, reduces the risk of rotation, swivel, or spread from one camera channel (320) to another. Thus, these features also help enable the seam 400 between cameras 320 to have a more consistent thickness than would otherwise have been the case with respect to the design values. The use of inner features (e.g., compensators, adjustment screws, and shims) within the lens housing and outer features between lens housings (e.g., channel-to-channel datums, ball and socket datums, and channel loading supports) assist in controlling core FOV or extended FOV aiming so that one camera channel can be aligned with an adjacent channel. The combined use of channel-to-channel datums, ball and socket datums (FIG. 10), and channel loading supports (FIG. 17) can also help reduce the sensitivity of the device to mechanical or thermal loads.

[0080] FIG. 10 shows additional design options for improving the opto-mechanical design of the outer lens element 437, or compressor lens element, adjacent to the seam 400 in the improved multi-camera panoramic image capture device 300. Specifically, the edge of the outer lens element 437 can protrude beyond the upper or outer edge of the lens housing 430 such that two adjacent outer lens elements 437 of adjacent cameras 320 can be in near contact at the seam 400. If these outer lens elements 437 are manufactured at least in part from a somewhat compliant material, some degree of actual physical contact can be enabled. However, if these outer lens elements 437 are manufactured from a brittle material such as glass, even greater care is required.

[0081] In the configuration of FIG. 10, two adjacent outer lens elements 437 having protruding edges are in close contact at the seam 400. In a preferred design approach, the opposing edges have a stepped edge angle 365 or structure. In the outermost portion, the two lenses and the housing can result in a parallel seam 400 with a width of 1.0 mm or less.

[0082] In the innermost portion where the inner beveled edges 370 of the adjacent outer lens elements 437 approach each other, a lens housing 430, an inter-channel datum 335, and a flat surface datum 670 can be provided. As a result, the edge of each outer lens element 437 has a stepped edge groove 380 that can be filled with a compliant adhesive. On top of that, along the edge 432 or seam 400, the outermost edge portions of the adjacent outer lens elements 437 spread, and these lens elements can be substantially adjacent, separated by a gap that can be less than 0.5 mm wide. In practice, the optimization of the seam width can depend on how brittle or compliant the lens material is (glass or polymer), the flexibility of the adhesive filling the seam, the use of other protection means, and the application.

[0083] FIG. 17 provides further details regarding how these and other features can be used during alignment and assembly. Specifically, FIG. 17 shows a partial cross-sectional view of an improved multi-camera panoramic image capture device 300 of the present approach, in which eleven camera channels 320 are attached to a strut or vertical central support 325 through which both wire wiring and cooling can be provided (see FIG. 15 for further details). FIG. 17 specifically shows the assembly without lens elements but indicates the design for opto-mechanical hardware (see FIGS. 9 and 15 for illustrations of opto-mechanical hardware including lens elements). FIG. 18A shows even more extremely detailed the main elements or components used in this alignment and assembly approach example.

[0084] As shown in FIG. 17, the top camera 320 can be identified as a primary alignment channel 610, while all other camera channels 320 shown are identified as secondary channels 615. Specifically, with respect to the primary channel, a ball pivot arm 342 having a ball pivot 340 of a channel centering hub 330 engages within a ball socket datum 356 of a ball socket array 346, where it interacts with a locking shaft retaining pin 348 that slides into a recess between the ball pivot arm 342 and the ball pivot 340 to stop Z-axis (vertical) translation. An anti-rotation keying pin 349 is press-fitted into a local hole provided within the ball pivot arm to stop the primary camera channel 610 from rotating about the z-axis. The ball pivot arm 342 of the primary camera channel 610 is lightly press-fitted within its ball socket datum 356 to prevent rotation about the X and Y axes.

[0085] Alternatively, the ball socket 345 may have a latching mechanism (not shown) for loading the ball pivot 340 into the ball socket datum 356 as a means for providing both a load force and a mechanism with reduced sensitivity to mechanical load forces applied to the camera or device. For example, the latching mechanism may have a latch that operates with an associated assembly. When the camera channel 320 or device 300 is affected by load forces, the latching mechanism is more compliant, robust, and reliable compared to the use of retaining pins. It should also be noted that the lens housing 430 may also have identification marks to facilitate alignment with adjacent lens housings of the housing.

[0086] As shown previously in FIG. 9 and shown in more detail in FIG. 18A, each camera channel 320 has a pair of inter-channel datums 335 on each face, which are small (local) and nominally centered along the face or side of the lens housing. For a pentagonal lens, each of the five sides may have a pair of datums (335). These datums comprise a curved surface 373 (shown in bold and extended for emphasis) to enable a two-point type of contact (375). Specifically, each pair of inter-channel datums 335 has a side that functions as a lateral datum between camera channels. They are bent to accommodate relative angular movement between channels. Only each side datum has an effective single contact 375 where each of their engaging datum radial surfaces meets. Since the radius is very large, the datum surface approaches a straight line. Thus, any shift of a camera channel relative to another that can be an angular or centerline offset will only have a minor effect on the relative lateral offset of a given camera channel as a result.

[0087] As further shown in FIG. 17, the improved multi-camera panoramic image capture device 300 may also have a channel loading support 630 that biases all secondary channels 615 with respect to the primary channel 610. The channel loading support 630 employs a pair of peripheral datums that are nominally identical to the inter-channel datums 335 used on each channel. The channel loading support 630 also has spring elements 635 that facilitate the loading of the secondary channels 615 with respect to the primary channel 610. Although the use of keyed supports can impose excessive constraints, the channel loading support 630 can also utilize a key feature for anti-rotation.

[0088] The primary alignment channel 610 (see FIG. 17) is aligned and locked in place when its ball pivot arm 342 and ball pivot 340 engage the pins (348, 349) of the channel centering hub 330. The secondary camera channel 615 is then added and its ball pivot 340 is fitted against the datum in the socket of the channel centering hub 330 and loosely aligned. When the channel loading support 330 is added at the seam 400, it lightly presses the secondary channel 615 against the primary channel 610 and against each other. The relative size of the ball socket 345 for the ball pivot 340 of the secondary channel is provided such that the secondary channel 615 is restricted in the Z direction only by the ball socket 345.

[0089] Next, at joint 400, for any two faces of adjacent camera channels (610, 615) to be parallel to each other, the configuration requires that at least three camera channels have their inter-channel datums 335 contact, at contact points 375, the opposing inter-channel datums 335 and their contact points 375. This effectively constrains the secondary channel 615 with respect to three degrees of freedom (DOF). Applying conventional kinematic machine design principles where motion and constraint in six DOF can be tightly limited with little crosstalk is essentially difficult in a device where a number of complex polygonal faces (e.g., pentagonal or hexagonal) can be placed very close together. In cases where four or more camera channels are very close together, if they are not in contact with each other, the camera channels can become overly constrained. As a result, the system can be only pseudo-kinematic, and mechanical stress and strain can then cause component misalignment or damage. These potential problems can be overcome in various ways. As an example, the relatively small size and centering position of the inter-channel datums 335 can limit any angular or spatial misalignment to a relatively small amount. As another example, a compliant material such as RTV can be applied to joint 400 to absorb any stress or strain, while the lens housing 430 can be designed and manufactured to be sufficiently rigid to resist deformation, misalignment, or damage. Joint width variations, including those caused by dynamic ones due to thermal or mechanical loading, can also be corrected by providing an improved camera 320 that captures image light 415 in an extended FOV 215.

[0090] As shown in FIG. 18A, the contacts can occur along a line that crosses the radial surface of the datum. Thus, each inter-channel datum 335 contacts the set of adjacent inter-channel datums 335 in a “line” contact when the engaging surfaces are preferably parallel. Otherwise, only point contact occurs. Typically, the peripheral inter-channel datum 335 contacts the opposing surface at only a single point. For each pair of inter-channel datums, only a single datum can contact due to dimensional differences. This is an advantage of this design approach since only a single datum can affect the gap or seam 400 between adjacent camera channels. This approach can be used in various designs for an improved multi-camera panoramic image capture device 300 that includes a device having a “soccer ball” or truncated icosahedron (FIG. 4) geometry, in which case the primary alignment (camera) channels can be the outer hexagonal lenses. This approach can also be used in the structure of a “hemispherical” device (see FIG. 2).

[0091] FIG. 18B-1 shows a cross-sectional view across an alternative or improved version of the seam of the structure of the lens housing 430 and their junction portions near the seam 400 as shown in FIGS. 10 and 18A. Specifically, FIG. 18B-1 shows portions of two adjacent lens housings 430 disposed around the seam 400, each housing supporting an outer compressor lens element 437 and at least one second compressor lens element 438. The lens housing 430 includes an inner light trap 457 and side surfaces that extend into a groove 433 cut into the edge 432 of the outer lens element 437. The outer wall of the lens housing 430 is tapered and flares towards the center of the device to provide greater mechanical rigidity and robustness. The interaction of two pairs of adjacent inter-channel datums 335 can be seen in the cross-sectional view.

[0092] FIG. 18B-2 shows alternative cross-sectional views of these same components, cut along or within seam 400, to illustrate the interaction of the inter-channel datums 335. As in FIG. 18A, adjacent inter-channel datum pairs 335a and 335b each have a curved surface 373 that locally interacts to assist in aligning or juxtaposing adjacent camera channels. In this example, to further reduce the potential for overconstraint, the inter-channel datum pairs are asymmetric and partially offset, thus highly likely to provide local point contact and less likely to cause overconstraint. The round holes 530 provide access through the sidewall of the lens housing 430 to enable the application of adhesive or RTV used in the attachment of the outer lens element 437 to the housing.

[0093] During assembly and alignment of the camera channels, it is also important to properly position the lens element within the housing. For example, the compressor lens (437) may have datum features along edge 432 that interact with a set of mating datum features on the inside of the lens housing 430. The datum features provided on the lens element are either features that are reasonably machined or polished with the intention of limiting vertical and concentricity errors, or are designed to be attached with adhesive on the beveled surface of a polished glass lens. As shown in FIG. 16, these features may include flat datum features 650 fabricated at the corners on the bottom surface of the compressor lens, outside the FOV. Other datums, including adjacent flat edges 660 that can be polished on the round lens element before it is formed into a pentagon with a truncated or beveled edge 370, can engage with the flat surface datum 670 on the lens mount. Since the glass is typically attached to the metal with adhesive, these flat edge datums can provide inductive alignment without the risk of overconstraint. The outer lens element 437 may also include an anti-cocking datum feature 680, perhaps attached to one of the beveled surfaces.

[0094] Due to these datum features, chamfers or tapered finishes on the pentagonal lens surface cannot be manufactured very accurately, thus helping to reduce lens costs. Even though FIG. 9 shows a pentagonal compressor lens or outer lens element 437 and housing 430, these mechanical approaches for reducing alignment errors are also applicable to hexagonal lenses, or lens elements having other polygonal shapes.

[0095] Alternatively, or in addition, the lens housing 430 may include one or more tabs or strut-like structures (not shown) that project out of the lens housing 430 from the nominal conical space or volume and can interact with a similar protruding structure of an adjacent lens housing or, alternatively, with a recessed structure of an adjacent lens housing. For example, these protruding structures can be provided generally proximate to the sensor 270 and the sensor package 265, are a few millimeters in length, and have datums features for kinematically controlling the DOF, such as a radial position from the center of the device or an inclination or rotation from one camera channel to an adjacent camera channel. The camera assembly can have two such protruding structures arranged symmetrically or asymmetrically around the lens housing and oriented orthogonally to each other to control different degrees of freedom. Alternatively, or in addition, the camera channel can have a lens housing 430 that includes one or more protruding tabs or strut structures disposed within the seam 400. For example, such a datum feature (not shown) can be provided within the seam at the apex 60 on the polygonal outer surface of the camera channel, project out of the nominal conical space or volume, and enter the seam between two adjacent channels. Depending on the device design and intended use, the protruding tabs or structures can be made of a compliant or rigid material, or a combination thereof, whether disposed within the outer seam 400 or more deeply embedded, such as near the image sensor. As another alternative, one or more tabs of a given lens housing 430 need not project outside the nominal conical volume or frustum, but a clamp that bridges from one tab to a tab of an adjacent lens housing can provide a joint or control to limit the degrees of freedom.

[0096] FIG. 19 shows an alternative version of the joint of the camera channel to the mounting structure near the center of the device, compared to that of FIGS. 9 and 17. Specifically, FIG. 19 shows a portion of two adjacent camera channels, the upper primary channel 610 in cross-section, and the secondary channel 615 in partial perspective view. In this case, the tripod or channel centering hub 330 has a socket 545 that provides a concave surface that nominally contacts the mating convex surface of the central hub 550. The central hub 550 is part of a mounting mechanical assembly that is nominally positioned with its center at the center of the device, and it can be attached to the support strut 750. The cable 560 with a ball at one end can be used to pull or pin-tension the socket 545 against the central hub 550. Each of the primary channel and the secondary channel can be held together by a similarly tensioned cable that goes down into the support strut, where they are fastened and locked. Alternatively, the primary channel 610 can be held in place with tightened bolts (not shown). Compared to the previous approach of FIGS. 9 and 17, this approach exchanges multiple balls and sockets for a reverse configuration with multiple sockets 545 (one per camera channel) that contact one main ball or hub 550. The tensioned cable replaces the previous approach that used retaining pins, or latches, or springs. The approach of FIG. 19 can allow multiple camera channels to be simultaneously and securely pulled around the center of the device in alignment with the ball hub 550. The ball hub 550 can be machined from a precision ball bearing.

[0097] Note that in the assembly approach shown in FIGS. 9 and 17, with the primary channel 610, and the secondary channel 615, and the centering hub 330 that interacts with the central ball socket array 346, the available space within the center of the device, where the power and communication cables, the cooling tubes, and the support mechanism need to fit within it, can be tight.

[0098] As shown in FIG. 13B, an improved camera 920 having a track length 980 between the front lens center and the imaging plane 950 can be positioned at an offset distance 925 from the imaging plane 950 to a low parallax amount 992. As one approach to improving device center congestion, the camera 920, its housing 430 (not shown), and the improved device 300 as a whole can be designed to provide an axial center offset distance 915 along the optical axis 985 between the low parallax amount 992 and the device center 910, similar to that described above with respect to FIG. 5C. Designing within the offset distance 915 (e.g., 1 - 4 mm) can provide extra space for power, communication, or cooling connections, cables, and mechanical support for the sensor package 952, or for other structural members. In the example of the camera system 920 shown in FIG. 13B, the improved low parallax multi - camera panorama capture device (300) can have a dodecahedron shape, and as a result, the device center 910 is the center of the nominal dodecahedron polygon structure. This offset distance needs to be determined during the design process of the camera 920 and the device 300 as a whole, as it interacts with the optimization of the lenses near the edge of the FOV. Thus, this optimization can be determined by or interact with the seam width, distortion correction, front - color control, reduced parallax for peripheral rays (edge rays 972) or imaging rays 975 in general, or the range and optimization of the LP amount 992 and the lower structure, the sizing of the lens elements (especially for the compressor lens group 955), or the tolerances for the extended FOV 215.

[0099] As another option for providing further access to cable wiring, support, and thermal management hardware, the ball and socket approach of FIG. 17 can be replaced with an internal frame of a polygonal face (FIG. 20) with access holes to a hollow center. For an improved multi-camera panoramic image capture device 300 constructed in an icosahedron pattern, the internal frame can also be an icosahedron with pentagonal faces, and can be oriented with internal pentagonal faces nominally aligned with the external pentagonal geometry. The internal frame can be machined separately and assembled from two or more pieces, or can be made as a one-piece structure by casting or 3D printing. The manufacture of a one-piece frame is more complex, but the resulting structure is more rigid and robust and can support tighter mechanical tolerances. For example, an icosahedron frame with a hollow center can be cast in stainless steel and then selectively machined post-casting to provide precision datum features. This internal frame can then have flexures or adjusters provided over all or most of the pentagonal faces to provide kinematic type adjustments to reduce or avoid overconstraint during device assembly and use. As described above, the adjusters available on these inner faces can be different for the secondary channels compared to the primary channels. Alternatively, the internal frame can be made at least in part of a more compliant material such as brass or invar. Since the central volume of this internal frame can be at least partially hollow, space can thus be provided for electrical cable wiring, thermal management hardware, and other support structures.

[0100] Figure 20 provides an example of such an internal frame 800 with a number of pentagonal faces 810 arranged in a dodecahedron pattern with a hollow center. The internal frame 800 has support struts attached at the 12th positions (similar to FIGS. 9 and 15) and can be designed as a mounting mechanical assembly for 11 camera systems. The polygonal internal frame, or half or partial internal frame, can also be used in a partial or hemispherical system, in which case a camera assembly, including an image sensor, is attached to the frame. Alternatively, a hemispherical system with an internal frame 800 (see, e.g., FIG. 21) uses a central hollow space (e.g., connection) to allow image light to pass through, including through an intervening relay lens system (725), and reach an image sensor on the other side. As shown in FIG. 20, one pentagonal face (810A) can have three adjusters 820, such as a set of screws or flexures, oriented nominally 120° apart, which interact with features on the camera housing and can thus be used to assist in aligning a given camera channel. As described above, the mounting and adjustment for the secondary channel can have a different design or configuration than that for the primary channel. As another alternative (not shown), one or more of the pentagonal faces 810A, 810B, or 810C can each include one or more adjusters that can be used to lightly press a camera channel against a precision V-groove structure (also not shown). These V-groove structures can be made within or project from the inner edge of the pentagonal vertices 60 of the pentagonal face. The internal frame approach can be used with other polyhedral device structures, such as for an icosahedron.

[0101] FIG. 11 shows a convenient hardware configuration in the area of sensor 270, cover glass 272, and an associated sensor package 265 (which may include an electronic device, cooling, and a mounting base). Cover glass 272 can seal or protect the sensor from the environment. Cover glass 272 can provide UV or IR cut filtering by means of a thin film interface or a dichroic coating, or that function can be provided on another window, an external filter 295. The UV or IR cut filter reduces the level of non-visible light incident on sensor 270 that is associated with image light 415. Alternatively, or in addition, UV and IR cut filtering can be provided by a coating applied to a lens element, including the outer surface of the outer lens element 437 (FIG. 9). The cover glass or filter 295 can also be UV light absorbing glass and can provide UV filtering by a combination of absorption and coating reflectivity.

[0102] In the provision of an improved multi-camera panoramic image capture device 300 with an inner beveled edge 370 (FIGS. 10 and 16), it has been recognized that the centering tolerance can be worse compared to a typical lens element manufactured with a conventional cylindrical edge. Such eccentricity can, as a result, affect the centering of the FOV captured across the camera lens system 320. As one approach for such error correction, the positioning of the effective image centroid or the center pixel can be determined either optically or electronically (FIG. 11). As another approach, an improved lens housing 430 can be designed to provide corrective lateral adjustment for one or more internal lens elements. For example, means can be provided to adjust the positioning or tilt of an intermediate internal lens element, such as a lens disposed between the aperture and the image sensor. The adjustment means can include or can use a micrometer, pins, shims, flexures, or springs. A Z-axis compensator for adjusting the focus or magnification difference can also be provided within each camera channel by a similar mechanism. The mechanism for enabling correction can be incorporated within the device or camera, or can be internal, external, or a combination thereof.

[0103] The opto - mechanics of FIGS. 8 - 10 and FIGS. 16 - 20 can reduce alignment errors for sensor 270, camera housing 430, and one or more cameras 320. However, these design improvements, and others equivalent thereto, may not provide sufficient accuracy for all configurations or uses of the improved multi - camera image capture device 300. As another, or complementary, approach, an optical reference system can be provided. Specifically, as shown in FIG. 11, a light source 460 can direct light 462 within a camera 320 of the improved multi - camera panoramic image capture device 300 into a window or filter 295 for optical reference. The optical reference light 462 can be coupled to the edge of the filter 295 and propagated to an output coupler 465 by total internal reflection (TIR), where it is directed towards the sensor 270, providing one or more illuminated spots or regions that function as an optical reference 475. The optical reference light 462 can be low - output infrared (IR) light, such as 785 or 835 nm, and the output coupler 465 can be one or more small lenses, prism features, or diffraction gratings. The illuminated region that functions as the optical reference 475 can be a focal spot only a few sensor pixel widths wide. The optical reference light 462 can be provided by a light source 460 that is mechanically stably positioned with respect to the image sensor 270. The reference light 462, which remains within the optical substrate of the filter 295 and is directed towards the opposite edge, can be absorbed by an absorber 470, which can be provided, for example, by a black paint coating.

[0104] FIG. 11 illustrates one advantageous approach for providing an optical reference 475 that enables FOV adjustment for camera 320 and thus helps to limit parallax errors and image overlap or underlap for adjacent cameras within the improved panoramic multi-camera capture device 300. However, generally, an optical reference can be provided by a light source disposed proximate to the sensor. As another example, the light source can be mounted on or near the sensor surface to direct light outwardly toward or through the cover glass such that the light reflects off the optical element and back toward the image sensor. The light source can be positioned outside the active area of use of the sensor, such as with a reflective coating. The reflective coating can be a local metallic or dielectric coating positioned outside the aperture used by the passing image light. As described above, the light source can then provide at least one spot of light irradiation on an active sensor pixel that is outside but proximate to the active area of use by the image light, thereby providing the optical reference 475. This concept can also be extended such that the optical reference can be attached to or interact with other components within the system, including lens elements or lens housing structures. Thus, the relative movement of a particular lens element or subgroup thereof can be monitored to provide information for image tracking, trimming, or correction efforts. Additionally, if one or more lens subgroups or compensators can be actively driven, such as using a motor, the resulting data can be useful to provide information for these corrections.

[0105] As shown in FIG. 12, the camera 320 for the improved multi-camera panoramic image capture device 300 may also include a mask or an internal baffle 455, which can be disposed between the outer lens element 437, or the compressor lens, and the inner lens element 440 behind it. As suggested by FIGS. 9 and 16, the sloped edge 370 of the outer lens element 437 can be manufactured through a curved outer lens surface at a distance set from and parallel to the nominal marginal chief ray 170 that enters along the straight outer edge of the regular pentagon. The sloped edge 370 can have a curved bevel shape or profile. However, if the polished edge (370) has a straight bevel or chamfer, it is easier and less expensive to manufacture the outer lens elements 437 and mount them in the lens housing 430. However, this then means that the volume of the optical glass along the straight edge of the pentagonal (or hexagonal) outer lens element 437, which can accept light that can become stray light, i.e., image light that complicates image mosaicking or tiling, varies along the sloped lens edge. The baffle 455 provided in FIG. 12 can also provide a sharp polygonal edge opening (e.g., pentagonal or hexagonal) that follows the shape of the outer lens element 437 and the core FOV 205, as well as a blackened surface for blocking and absorbing light outside the intended FOV. Alternatively, the light-absorbing baffle 455 can be printed or coated on the inner lens element surface. Thus, the baffle 455 or mask also defines the edge of the passing image light and can thus cast an edge shadow on the image sensor 270 (see FIG. 11).

[0106] The shadows 495 cast by the optical reference light 462 and the baffle 455 can be used both individually and in combination to result in an improved multi-camera capture device 300 with reduced parallax error and reduced image overlap or underlap between cameras. As shown in FIG. 11, incident image light 415 incident on an image sensor having an active area length and width can provide an illuminated image area 450. For example, a camera 320 that collects uniform image light within a nominally pentagonal core FOV 205 can, as a result, provide a pentagonal illuminated area 450 on the sensor 270. This image or illuminated area 450 can underfill, more or less, the width of the sensor 270, depending on the shape of the illuminated area and the sensor. For useful data, all or most of the projected penumbra 495 can define the boundary of the illuminated area of the active pixels within the underfilled sensor area. In some examples, the penumbra 495 fills most (e.g., ≧97%) of the active area width. The baffle 455 operates like a field stop. In a camera 320 without a baffle, the outer lens element operates like a field stop to define a pentagonal FOV.

[0107] Within the illumination region corresponding to the penumbra 495, there is a smaller illumination region 450 corresponding to the core FOV 205, and there may be an intermediate extended FOV 215 therebetween. The extended FOV 215 may be large enough to be substantially consistent with the size of the penumbra 495. The difference can be optimized according to the mechanical alignment tolerances expected for the positioning of the lens assembly and the baffle 455, as well as the optical sharpness or transition width of the projected shadow. The underfill of the sensor active area width enables the core FOV 205, the image center of gravity, the extended FOV 215, and the penumbra 495 to shift their positions on the sensor without being cropped. During the calibration process, it may then be beneficial to establish the image center of gravity 480, which can then be tracked with respect to time and exposure conditions. Specifically, the initial distance or offset 485 can be determined during the calibration process and stored as a pixel count. As a result, during or after the assembly of the camera 320 into the multi-camera capture device 300, if the illumination image region 450 shifts, whether due to mechanical or thermal reasons, the new position of the image center of gravity 480 and the shape, size, and position of the FOV 205 can be determined and compared with the previously stored calibration data.

[0108] However, at the completion of this process, it is necessary to determine the boundaries of the illumination region 450 corresponding to the desired image data. As suggested in FIG. 7, the penumbra 495 cast by the baffle 455 can then provide a useful series of reference edges or points 210 that are close to but slightly larger than the core FOV 205. The shape and position of the shadow or occlusion cast by the baffle 455 can be determined during the bench test of a given camera 320 prior to its assembly into a given multi-camera capture device 300. Similar calibration data can be obtained after its assembly, and then over time as the same camera and device are used, changes in the positioning of the shadow that may occur due to environmental or mechanically initiated changes in the internal mounting or positioning of the lens elements (e.g., 437, 440) or the baffle 455 can be tracked.

[0109] More specifically, the FOV edges that define the baffle 455 and the optical reference 475 (see FIGS. 11 and 12) are used in combination to monitor or longitudinally track the position of the image centroid 480 and the image region edges, which can assist in image mosaicking or tiling. Specifically, for the purpose of aligning the camera lens 320 both centrically and rotationally with respect to the image sensor 270, methods of projecting an optical reference or casting an optical occlusion onto the image sensor 270 can be used either individually or in combination.

[0110] Essentially, the baffle 455 or mask casts a shadow with multiple edges onto the sensor 270, and the shape of the baffle is sampled from around the periphery of the projected image or illumination area 450 captured by the sensor 270 either in software or hardware. The edges of the shadow are relatively sharp but can still create a gradient fall off, potentially spreading at least some pixels between the illumination area 450 and the dark periphery of the projected shadow. Data for the edges or references sampled from around the masked area and the edges of the shadow gradient can then be used to derive the image centroid 480 of the lens projection. The baffle 455 or the masking shadow 495 can also have additional features for indicating rotation if needed for derivation. The calibration step can first be used to derive the relationship of the shadow 495 to the lens center and rotation. Additionally, the dimensions and shape of the mask are accurately measured prior to installation and can be compared to the size and shape of the projected shadow 495. The derived relationships of the size, shape, and centroid of the projected shadow 495 can absorb the expected or measured differences in the shape and size of the edge shadow 495 compared to the shape and size of the baffle. Other factors that can affect the edge shadow 495, such as the tilt of the mask and the effects of lens distortion in the partial field of view portions (e.g., 0.85 - 1.01) corresponding to the outer portions of the core FOV 205 and the extended FOV 215, can also be absorbed.

[0111] Additionally, as described above, the optical reference 475 can be projected onto the unused portion of the sensor 270 using IR or visible light. The pattern formed by the reference can then be used to derive the position of the reference and to calibrate the image centroid 480, as well as the lens center and rotation. The calibration step is initially used to derive the relationship of the optical reference 475 to the lens center and rotation and is associated with the distortion characteristics calculated in the calibration process. Additionally, a series of calibration images of the shadow 495 of the reference provided by the baffle 455 can be used to find more distant features (e.g., corners) and thus confirm or determine the planarity of the lens with respect to the mechanical alignment of the sensor. The sensor alignment or planarity can be corrected using the adjustments described above with respect to FIG. 8. The combined reference method, using both the optical reference 475 and the projected shadow 495 of the reference, has the advantage of being more robust in various lighting conditions where the edges of the shadow casting method may be inconsistent or difficult to detect.

[0112] Methods for calibration with an optical (or electronic) reference and shadow masking can be used to accurately align the modeled lens distortion derived from the calibration step to the calibrated core FOV 205 for a given camera. The generated correction data can be stored in matrix or other form in an on-board look-up table (LUT) on the local substrate that is part of the sensor package for each camera and sensor. When an image is captured, this correction data can be used to trim the large image corresponding to the extended FOV 215 to an image corresponding to the actual current core FOV 205 during the initial or intermediate capture step. Similarly, other available data, such as for exposure and color correction, that can be stored locally can also be applied in real time to the trimmed, core FOV-sized image before it is sent to the system computer for further image processing, including stitching or tiling of the images into an assembled panoramic image.

[0113] Looking more broadly, generally, during either real-time or post-processing steps of modifying image data captured by one or more cameras 320 of an improved multi-camera panoramic capture device 300, a method for calibrating optical references and shadow shielding can be used to accurately align the modeled lens distortion derived from the calibration step to the captured image for a given camera. The resulting image can be accurately undistorted as needed and also corrected to account for other camera intrinsics (e.g., lens focal length, or sensor parameters). This method can be iterated for multiple cameras 320 of the multi-camera capture device 300 to enable accurately mosaicking or tiling multiple images together. By accurately fitting knowledge of camera intrinsics to the captured images, the quality of the mosaicking across the boundaries or seams between images captured by adjacent cameras improves the combined image quality and reduces errors from images that are not correctly aligned to the initially calculated calibrated intrinsics. Thus, the speed of mosaicking or tiling increases because computationally intensive (e.g., optical flow) image feature-based methods for correcting misalignments, distortions, and lens intrinsics in alignment are hardly, or not at all, time-consuming.

[0114] FIG. 11 also shows that the outer portion of the rectangular image sensor 270, outside the active or pixel area, can be covered or shielded by a mask 455. In cases where the sensor 270 is much larger than the image illumination area 450, the use of the mask 455 prevents stray light from entering the lower dark area that would otherwise occur. As a result, these pixel values are essentially always "zero", and their outputs can be ignored or dumped, which speeds up the image processing and compression time. Also, the light source 460 and the optical reference 475 can be exchanged or replaced with an offset value 485 relative to an electronic reference calculated as the difference at the calibrated position of the center of gravity 480 relative to the sensor edge or the mask edge. The use of an electronic reference can be simpler and less expensive to implement compared to the optical reference 475, but depending on the mechanical design of the camera housing 430 or the use case for the improved multi-camera capture device 300, it may or may not be mechanically or functionally robust.

[0115] Differences in brightness for each scene or each camera can affect the image content incident within the illumination area 450 corresponding to the core FOV 205. Specifically, image quality and image mosaicking or tiling vary depending on the content, such as a dark scene for one camera and a bright scene for an adjacent camera, and can affect image mosaicking and exposure levels. These differences not only make it difficult to detect the edges of the image or shadow, but also make it difficult to determine the image center of gravity values used during image stitching. Additionally, these exposure variations can cause the mosaicked image to appear tiled along or across the edges. Either an electronic or an optical reference can be used to assist in determining the image center of gravity and the image edges, but the optical reference can also be used to enable electronic exposure control correction. Additional exposure correction can be provided by matching the center or average values of adjacent cameras and scenes to assist in better mixing or harmonizing the mosaicked images.

[0116] Mask 455, shown as part of the sensor portion of camera 320, can be removed or reduced if sensor 270 has a configuration close to that of core FOV 205 or the shape of image illumination region 450. Replacing a rectangular sensor with a square, or a substantially square alternative, can more effectively utilize the available space. Specifically, only a much smaller portion of the conical volume, which the camera lens assembly could nominally fit within, is lost to support the unused sensor area. The illumination region 450 can be placed on a sensor with a more appropriate active area, and thus the effective resolution (pixels in the sensor, or pixels / degree with respect to core FOV 205) can be improved. However, the use of a square image sensor can still provide space or pixels for supporting the use of optical reference 475. Alternatively, a rectangular image sensor can be replaced with a sensor having an active area optimized for the camera shape, such as having a hexagonal or pentagonal shape. Having a shape-optimized sensor can make the optical design easier, and thus make it easier to fit the camera assembly within the nominal conical volume or frustum and optimize the "NP point" position and size. The use of a shape-optimized sensor can provide greater freedom in the trade-off or balance of optical design optimization for factors including the relative distance between the image sensor and the NP point, image quality, coma, image distortion, and ray constraints for reduced parallax. As a result, the optical resolution (e.g., from the lens) and angular resolution (from the sensor) can be further optimized to exceed 100 px / degree. However, the design can still underfill the core FOV 205 with the optimized shape sensor to allow for margin for image capture in the extended FOV and avoid potential loss of image content due to image shift and rotation. Underfilling, if large enough, can also allow for space to provide the optical reference 475.

[0117] Note that for many image sensors 270, the image light is incident directly on the sensor pixels. However, many commercially available image sensors further comprise an integrated small lens array that overlaps and is aligned with the pixel array. A given small lens directs a portion of the incident image light towards the corresponding underlying pixel. This approach can reduce optical crosstalk between pixels because the incident image light is more likely to be photoelectrically converted into an electronic signal within the incident pixel than might otherwise be the case. Since the pixels are innovatively small, this approach is becoming increasingly common as a means to improve or maintain the image resolution (MTF). Such sensors with an integrated small lens array can be used in the cameras 320 of an improved multi-camera capture device 300.

[0118] Alternatively, the overlapping small lens array can provide an array of small lenses, where any given small lens directs light onto a plurality of image sensor pixels. In this case, direct image resolution may be lost, but there is the benefit of acquiring light field capabilities to provide either stereoscopic image capture or increased depth of focus image capture. This type of light field capability can be designed into a panoramic multi-camera capture device 300, but much of that benefit may then be lost in the FOV overlap (Figure 3). In such a system, the optical design may also have an entrance pupil in the front or outer lens element, which can cause information loss. However, incorporating the light field capability into an improved panoramic multi-camera capture device 300 with a plurality of adjacent cameras 320 with reduced parallax error, regardless of an improved kinematic design, an improved opto-mechanical design at the seam 400, or the use of electronic or optical references or edge masks, can improve the performance and value of the light field data captured for such a system. In particular, in such a system, a camera 320 with light field imaging can capture a portion of the plenoptic light available to provide an image with improved depth of focus and object perspective without either information loss due to parallax error or complex image fusion occurring.

[0119] As described above, the seam width affects the parallax error, the size of the blind area 165, the size of any complementary image overlap (Figure 3), as well as the image processing and the mosaicking or tiling time, so the seam 400 can be an important parameter in the design of the improved multi-camera capture device 300. For example, in cases where the expensive multi-camera capture device 300 is used in a controlled environment, a narrow seam with a small tolerance may be possible or acceptable. However, local collisions or stresses of expensive brittle materials (e.g., glass) can result in delamination and other damage and should be avoided. Needless to say, there are other external damage risks, including those from external materials in contact with the outer lens element 437. Therefore, since the multi-camera capture device 300 can be an expensive unit, an optomechanical design that provides risk reduction or protection for the camera or device can be beneficial.

[0120] As one solution, the seam 400 can be filled with a compliant material such as an RTV or silicone-based adhesive. As another solution, the optical design of the camera lens can include an outer lens element 437 designed as a polymer or plastic material that is less brittle than glass, such as Zeonex. The combined use of the polymer outer lens element 437 and the seam 400 filled with a compliant material can further reduce the risk.

[0121] However, in optical design, glass usually provides better and more predictable performance than polymer materials. As one approach, the outer lens element 437 can be designed using a common, more robust, and less expensive optical glass such as BK7. The outer lens element, regardless of whether it is glass or polymer, can also be overcoated with an anti-scratch AR coating and an oleophobic or hydrophobic coating. The compliant seam filling adhesive should likewise resist penetration or contamination by water, oil, or other common materials.

[0122] It may be advantageous to design a camera lens 320 having an outer lens element 437 or a lens element 435 including a compressor lens, which is more of a meniscus type lens element than that previously suggested (FIG. 2A). As an example, as shown in FIG. 13A, the compressor lens is divided into a compressor lens group including a first compressor lens element (437) and a compressor lens element 438 coupled as a doublet. As a result, the outer lens element 437 can be designed more freely to have a partially meniscus shape while still providing both some ray or image light compression or direction change towards a wide-angle lens group including a fourth lens element 442, and the main light collection for reducing parallax. If the outer lens element is more meniscus-like or has a generally long focal length, it can be made flatter and does not protrude much above the lens housing 430. Thus, it is also less likely to receive rays 410 at extreme angles from the object space 405 that can reach the image sensor 270 well as visible or detectable ghost light, except from outside the nominal field of view 425. In such a lens design, the second compressor lens element 438, or doublet, provides additional optical power to direct the passing image light inwardly towards the inner fourth lens element 442.

[0123] Additionally, the camera lens 320 can be opto-mechanically designed such that the outer lens element 437 can be a field-replaceable unit (FRU). One way to support this approach is to design the lens element to have a more meniscus-like shape. The opto-mechanical design can be such that the outer lens element 437 is enabled to be mounted so that it can be easily removed and replaced in case of damage. For example, the outer lens element can be directly mounted within the camera lens housing 430 using datums such as those shown in FIG. 16, but can then be removed using one or more extraction tools. An insoluble (desolvable) adhesive (e.g., glyptal) or a thermal adhesive having a transition temperature above terrestrial extreme can also be used. The replacement outer lens element can then be mounted in its predetermined position. Alternatively, the outer lens element can also be an assembly mounted to a sub-housing portion that can be removed from the main camera lens housing 430. In either case, providing a lens design where the outer lens element has a partial meniscus or telephoto design results in a reduced sensitivity of the design with respect to the image quality and image position on the sensor, leading to an inaccurate alignment of the replacement outer lens element. Potentially as a result, cast or molded optical elements can also be used, whether made of a polymer (e.g., acrylic) or glass (e.g., B270). Camera recalibration using the previously described optical or electronic references, or the shadow 495 of the baffle 455 further reduces the risks associated with outer lens replacement.

[0124] As another alternative to the improved multi-camera capture device 300, the entire camera 320 can also be designed to be modular or a potential FRU. In such a case, the camera 320, including the housing 430, and the accompanying lens elements including the outermost compressor lens element 437 can be removed and replaced as a unit. For example, the pressure from the channel loading support 630 can be released or reduced, the ball pivot of the camera channel can then be released from the ball socket 345, and the replacement camera channel can be inserted into its predetermined position. Thereafter, the pressure from the channel loading support 630 can be restored. Since the relative position of the camera channel can shift slightly during this assembly process, the process of FOV centering or calibration with a reference may then need to be repeated. Depending on the design complexity, FRU replacement of the modular camera lens assembly can occur in the field, at a service center, or at a factory, but relatively easily and quickly.

[0125] The improved multi-camera capture device 300, and the cameras 320 therein, may also be protected by a dome or shell (not shown) having nominal concentric inner and outer spherical surfaces through which the device can image. The addition of an outer dome can be used to surround the substantially spherical device of FIG. 15 within the internal volume, or for a substantially hemispherical device such as that of FIG. 21, or a device having an alternative geometry or total FOV. The dome can be composed of a pair of mating hemispherical or substantially hemispherical domes connected at a joint, or can be a single substantially hemispherical shell (e.g., for FIG. 21). The transparent dome or shell material can be glass, plastic or polymer, hybrid or reinforced polymer material, or a rugged optical material such as ceramic, sapphire, or Alon. An optically clean dome or shell helps to eliminate environmental contamination and, similarly, can function as a FRU and be replaced if damaged. Replacing the FRU dome can be easier than replacing the entire camera 320 or an outer lens element or outer lens element assembly of the FRU type. The dome or shell can also be enhanced with an AR, oleophobic, or hydrophobic coating on the outer surface, and an AR coating on the inner surface. The use of a dome or shell can reduce the need or burden of using a carrying case or shipping container, although such an enclosure may still be useful. Alternatively, or in addition, the dome or shell can be faceted and provide a series of continuously adjacent lens elements that can function as outer lens elements for associated adjacent camera systems 320. This approach can have the potential advantage of reducing the width of both intervening seams 160 (e.g., seam width ≦ 0.5 mm) and their associated blind areas 165, allowing the device center 196 to coincide with a low parallax amount 188.

[0126] As yet another alternative, the improved multi-camera capture device 300 shown in FIGS. 14A and 14B, and the cameras 320 and their housings 430 therein, can be designed to provide a protective fin 510 protruding from the seam 400, or a protective strut 520 protruding from a vertex (60) or corner where multiple adjacent cameras contact. These fins 510 or struts 520 protrude into the object space 405 but can be designed to be outside the captured FOV by protruding less than the extent of the blind region 165, and they can prevent direct contact by any object from the outside environment. These fins or struts can also be designed to be sufficiently rigid with the correct aspect ratio of height to thickness to reduce the risk of their bending. Alternatively, the fins or struts can be designed with a somewhat compliant material to bounce back from the contact force, or a combination such as compliant fins and rigid struts can be used in parallel. The struts and fins can be supported by a compliant mounting base embedded within the seam 400, so that these protective features can return towards their originally designed shape without damaging the camera 320 or the multi-camera capture device 300 during an impact event. The use of dark-colored fins protruding perhaps only a few millimeters or tens of millimeters from the seam 400 can also provide the benefit of reducing the acceptance of light rays at extreme angles of incidence with respect to the camera lens, which can result in ghost light that can reach the image sensor 270 well. The use of fins or struts can protect the camera 320 and the multi-camera capture device 300 from only moderate forces or impacts, and thus will be of sufficient value for certain designs and applications. These fins of FIG. 14A or struts of FIG. 14B can also be sacrificial elements or removable FRUs themselves. Once they have performed their job or are damaged, they can then be replaced by inserting a replacement (e.g., perhaps into a receptacle socket specifically designed for that purpose).

[0127] The proposed width of the seam 400 in the previous example given was rather narrow. However, for the sake of clarity, in relation to the design and performance of the multi-camera capture device 300, it may not be necessary to hold the separation between the beams to exactly zero with respect to the chief ray 170 passing through the adjacent cameras 320 and for a given object distance. Rather, the goal may be to have a maximum of one pixel of unclear information (or, depending on the application, several pixels) at the object distance of interest. Depending on the allowed seam width and pixel loss, the cameras 320, and the multi-camera capture device 300 may be more or less robust, or more or less tolerant to both manufacturing tolerances and protection design approaches, including those described above.

[0128] Specifically, some manufacturing tolerances of the camera 320 (see, e.g., FIG. 9), the housing 430, the outer lens element 437, and the seam 400 can be tolerated or accommodated. The use of some optical or electronic references, or the shadows cast by baffles, allows these correction approaches to enable the collection of low-parallax error images from adjacent cameras 320 with a slight FOV overlap, since a "centered" image can be found. However, the multi-camera capture device 300 can also be tolerant of some image loss. For example, for an object distance of 5 feet and a device where the camera supports a total resolution of 320,000 pixels (8k output equirectangular image), a seam 400 with a width of 1 pixel corresponds to a gap of 1.2 mm. If the improved multi-camera capture device 300 is designed to hold the mechanical seam at 3 mm after construction, the camera can be designed for a 1.5 mm gap while tolerating some FOV overlap (□□). The device can have a mechanical design that tolerates a maximum □□ alignment pointing error with tolerances. After the assembly of the camera, the actual light rays collected along the edge surface shared between the camera lenses can shift, but always within the pointing error. Thus, the gap or seam 400 can be restricted to a maximum width of 3 mm. For some camera designs and markets or applications, wider seams can be tolerated, either in absolute size (e.g., 4.5 mm seam width) or in lost image content (e.g., 2 - 20 pixels per seam). The lost pixels can be corrected by increasing the FOV overlap or the extended FOV 215 between adjacent cameras to capture the overlapped content, but at the cost of some parallax error and some increase in the image processing burden. However, for a moderate amount of extended FOV (e.g., ≦ 5%), the residual parallax error (e.g., FIG. 8B) can still be moderate. If the seam 400 is smaller and the knowledge of the position of the image centroid 480, as well as the image size and shape, is better, the core FOV 205 on the sensor can be larger, and less lens performance and sensor area can be directed towards providing a larger extended FOV 215.

[0129] Seam 400 can be nominally the same width at the junction between adjacent cameras for all cameras 320 within the improved multi-camera capture device 300. Careful camera and camera housing design, as well as mounting brackets between cameras (Figs. 9 and 10) can assist this from occurring. Nevertheless, the seam width can vary either during device assembly or in a dynamic environmental state. For example, on one side of the first camera, the seam width between that camera and an adjacent camera can be only 0.75 pixel widths, while at the same time, the seam width between the first camera and another adjacent camera can be 3.25 pixel widths. The seam width can also vary non-uniformly. Images generated for the sensors can be shifted or rotated relative to expectations. Thus, such variations can increase the parallax error for images captured by these adjacent cameras, complicating image mosaicking or tiling. However, the use of electronic or optical references (475), or the shadows cast by the internal baffle 455, allows the image center of gravity and image edges (Figs. 11 and 12) to be monitored for each camera, enabling a quick reference to a nominally calibrated or expected state. This reference or correction data can also be compared from one camera to another for the purpose of defining for each camera the effective image center of gravity and image edges that most effectively minimize the parallax error, either individually or overall (e.g., averaged) for all cameras. This information can then be used during image processing to quickly and robustly determine the image edges, enabling efficient image mosaicking or tiling.

[0130] As described above, with respect to FIG. 15, a useful configuration for the multi-camera capture device 300 can be to design and manufacture a generally spherical system having a plurality of camera lenses distributed in an icosahedron or truncated icosahedron arrangement. However, a problem that can occur with an improved multi-camera capture device 300 of the type of FIG. 5 is that there is little room to include other components or functions due to the plurality of camera channels and respective image sensors nominally limited within a spherical shape. Thus, for some applications, a device having a generally hemispherical configuration with potential room below for other hardware can be valuable. However, since the outer lens elements and cameras are typically polygonal, the hemispherical device can have a jagged or irregular outer perimeter. Also, in such a system, one or more of the cameras can be designed using a fold (e.g., using a mirror or prism) such that the optical path extends through the bottom irregular peripheral surface. This structure can provide additional room for the use of modular sensors that can be swapped in and out.

[0131] However, for the "hemisphere" version of the truncated icosahedron, there are six camera channels with pentagonal faces and ten camera channels with hexagonal faces, and it can be difficult for the optomechanics to provide space for bending so many optical paths. FIG. 21 shows an alternative version of an improved multi-camera capture device 700 in which image light collected by each camera objective lens system 720 is directed along a nominally straight optical path and then through an image relay optical system (725) to a more distant image sensor disposed within a sensor housing 730. The original imaging plane provided by the objective or camera lens system is essentially the actual intermediate imaging plane within a larger optical system. It can be re-imaged at a magnification (e.g., 1:1 or 2.75:1) onto the next imaging plane (not shown) where the image sensor is located. Thus, conveniently, the image sensor can be larger and provide a higher pixel count, and the relay lens system 725 can re-image the image provided by the objective lens (720) at an appropriate magnification to nominally fill the more distant sensor with the projected image. FIG. 21 shows a semi-internal view revealing portions of two camera objective lens systems 720 with associated relay systems 725 each extending outside a nominally conical space or volume on the "hemisphere" portion of the truncated icosahedron. Image light from each camera 720 traverses a central volume or connection 710 as it passes through each relay lens system 725. The relay lens system 725 can assist in including the space necessary for the image light to pass, including a field lens (not shown), behind the imaging plane of the camera lens but before the connection. The optical system of the relay lens system 725 can also be designed in concert with the camera 720 to correct or compensate for its aberrations.

[0132] To provide a hollow central volume or connection 710 through which imaging rays from adjacent camera channels cross each other and pass through their respective relay lens systems 725, an internal frame 800 having a polygonal face (see, e.g., FIG. 20) has access holes to allow image light to pass through the hollow center. As previously proposed, the internal polygonal frame has flexures or adjusters provided over all or most of the polygonal faces to provide kinematic or pseudo-kinematic adjustment and to reduce or avoid overconstraint during device assembly and use. However, in this case, the hollow space or central volume of this internal frame is provided first to allow image light to pass through the central connection 710, and secondarily can also provide room for electrical cable wiring and thermal management hardware. Alternatively, a mirror (not shown) is also used in the relay lens path to redirect the image light so that the entire opto-mechanical structure is more compact. The improved multi-camera capture device 700 also includes a support structure 740, support struts 750, and cable wiring 760 for supplying power and extracting signal (image) data. The support structure 740 can provide more substantial support for the camera housing 730 than is illustrated in FIG. 21.

[0133] FIG. 15 shows an electronic system diagram for an improved multi-camera capture device 300. In this example, a dodecahedron type device has 11 cameras 320 and an electromechanical junction at the 12th camera position. Image data is collected from each of the 11 cameras, passed through an interface input-output module, through a cable or cable bundle, and can be directed to a portable computer that provides image processing including live image trimming and mosaicking or tiling, as well as camera and device control. The output image data is directed to an image display, VR headset, or a more distant computer located locally or remotely. Electrical output and cooling can also be provided as needed.

[0134] Also, as previously suggested, the performance of the multi-camera capture device can be affected by both internal and external environmental factors, in relation to both opto-mechanics and image quality. Each of the image sensors 270, and the sensor package 265 as a whole, can be a local heat source, together with the data interface and power support electronics. To reduce the heat impact on the camera lenses, and the images they provide, the mechanical design for the improved multi-camera capture device 300 can thermally isolate the sensors 270 from the lens opto-mechanics. To further assist in reducing the thermal gradient between the sensors and their electronics, and the optical system, a micro heat pipe or Peltier device can be used to cool the sensors and redirect the heat. The heat can be removed from the device as a whole by either active or passive cooling provided through the electromechanical joints at the twelfth camera position, as shown in FIG. 15. This cooling can be provided by convection or conduction (including liquid cooling) or a combination thereof.

[0135] As previously suggested, external ambient or environmental factors can also affect the performance of the multi-camera capture device. These factors can include the effects of ambient illumination, or thermal extremes or variations in the environment. For example, since sunlight is typically highly directional, certain scenarios of outdoor image capture can result in a plenoptic illumination from a scene where the cameras on one side of the device are brightly illuminated while other cameras are in shadow. In such cases, the captured images can exhibit dynamic exposure variations, which can then be corrected by exposure correction that can be provided locally (see FIG. 15). In the improved multi-camera capture device 300, the light from the optical reference 475 can also be used for exposure correction of the captured images. Light or pixel signals from a portion of the peripheral image region between the edge of the core FOV 205 and the extended FOV 215 can also be used for exposure correction before the image is trimmed down to the size of the actual current core FOV 205. It should also be noted that since the extended FOV 215 of the first camera can at least partially overlap with the extended FOV 215 of an adjacent camera, the light level and color comparison can be performed on the content or signals being captured simultaneously by both cameras. Signals or pixel data from these overlapping regions can be used to determine exposure variations between the two cameras by having a common reference point (e.g., using a matching feature point - SIFT, SURF or a similar algorithm to find common feature points within the overlapping region).

[0136] Note that peripheral images or exposure data can also be retained for later use in image post-processing. Additionally, exposure correction can also be enabled by embedding photodetectors within the seams 400, or at the vertices, between the outer lens elements 437. These rapid exposure differences can cause spatial and temporal differences in the thermal loading of some of the image sensors 270 within the multi-camera capture device 300 compared to others. The aforementioned sensor cooling can be designed to absorb such differences regardless of whether it is enabled by heat pipes, heat sinks, liquid cooling, or other means. Performance can be verified by finite element analysis (FEA).

[0137] Alternatively, one or more camera systems can be protected by attaching a shield or mask that covers the polygon from seam to seam and from vertex to vertex of its outer lens element. Such shields can be provided to cover a single camera lens system or multiple lens systems. These shields can be shaped to generally conform to the outer surface shape of the outer lens element and can be used to prevent saturation or overexposure from bright directional light (e.g., sunlight) or to prevent contamination from local directional environmental factors. These caps are nominally removable for some user applications, but they can be used continuously over a long period of time. Excessive bright exposure from the sun or other light sources can also be controlled by an electronic shutter or drain, or a physical shutter or electro-optical dimming filter, or a photochromic or electrochromic filter, designed, for example, within the grouping of inner lens elements 440, for example, within camera 320. Signals for initiating or controlling an electronic or secondary shutter can be obtained from the image sensor or from other internal or external light detectors. As another robustness improvement, one or more camera channels can use a dichroic color filter array integrated into the image sensor package instead of a standard dye-based CFA.

[0138] The environment's influence can also asymmetrically heat or cool the multi-camera capture device. The aforementioned nominal kinematic mounting or connection of adjacent camera housings 430 (see FIGS. 9-10 and FIGS. 17-18) to the improved multi-camera capture device 300 can assist in reducing this impact by attempting to deflect or average mechanical stress to limit mechanical movement. However, it may be further advantageous to provide channels or materials for transmitting or shifting an asymmetric heat load more evenly between the cameras 320 and their housings 430, or by the cameras 320 and their housings 430. With respect to FIG. 9, this means that spaces around the lens housing 430 and the channel centering hub 330, such as the inner volume 390, are at least partially compliant but filled with a high thermal contact, thermally conductive material (e.g., Sil-Pad (from Henkel Corporation) or CoolTherm (Lord Corporation, Cary, North Carolina)) that can assist in spatially averaging an asymmetric heat load or difference. Alternatively, or in addition, thermally conductive straps or tapes, such as the 88xx series of adhesive tapes from 3M (St. Paul, Minnesota, USA), can be used. However, at the same time, some of the effects of thermal changes can be mitigated with respect to the image performance of the camera lenses 320 by both a judicious selection of optical glass and athermal mounting of the optical elements within the lens housing 430. In combination, an effective design approach can be to enable thermal transfer or crosstalk between the lenses 320 and their housings 430 with respect to environmental influences, but at the same time separate the lenses and housings from the sensors 270 and their electronics.

[0139] An improved camera 320 for use within an improved multi-camera image capture device 300 can correct for focus changes (e.g., defocus) caused thermally or mechanically, also using an adjustable lens element. This adjustable lens can preferably be disposed between inner lens elements 440 and can be a liquid crystal or elastomeric polymer type device, such as an electrically actuated focus-adjustable lens from Optotune (Dietikon, SW).

[0140] Emphasis has been placed on the development of an improved camera 320 having a polygonal outer lens element for use within an improved multi-camera image capture device 300 to capture and image light from a polygonal FOV. A number of such adjacent cameras can be used in a nominally spherical or hemispherical device. However, a device 300 having a further reduced number of cameras and covering a further reduced total FOV can be developed. For example, a system with only 4 or 6 adjacent polygonal cameras with low parallax may be suitable for some market applications. Additionally, a single camera having a polygonal outer lens element that captures image light from a nominally matching polygonal FOV with reduced parallax or perspective error can be used alone, such as for security or surveillance applications. For example, a single camera designed opto-mechanically to fit within an eighth of an octahedron can be mounted in the corner of a room ceiling to capture the image content of the room environment with little or no blind area. Similarly, emphasis has been placed on the development of an improved camera lens system 320 in which parallax error can be reduced at least within the core FOV 205, although a moderate extended FOV 215 (e.g., an extra ≦ 5%) and image capture overlap with adjacent cameras can also be provided. Similarly, it should be noted that this approach can be extended to support conceivable applications having a further larger overlap FOV between adjacent cameras (e.g., an extra 10 - 25% or an extra FOV of about 4 - 10° for a dodecahedron system with a nominal core FOV of 37.45°), and at the same time having parallax error or perspective error control at least within the core FOV 205. The camera design can be further extended to provide a further larger overlap FOV (e.g., an extra 10 - 20°), but there is no benefit of reduced parallax for angles well beyond the designed core FOV.

[0141] As described above, this approach can be used to design low-parallax lens systems for various applications. The method of device development can be outlined, for example, as follows: 1. Define in terms of the customer or application space, polygonal device configuration, resolution, expected range of seams and blind areas, and minimum object distance. The minimum object distance is the shortest distance at which immediate image stitching or image mosaicking of low-parallax overlapping images can be applied if the image processing is acceptable. At this distance, estimate the maximum total gap or seam for which the resulting image loss or difference is generally imperceptible to the human eye. For example, for an 8K rectangular projection image, an expected image loss of 1 / 40% of the output image (along the equator) corresponds to a seam of 2 pixel widths each. An alternative minimum object distance can be defined that is not acceptable even for conservative image processing and is even further away. 2. Design the imaging lens while controlling parallax and front color. The design can be further modified to provide an additional or extended FOV where parallax can be controlled. This additional FOV can provide room for the rainbow tinting of the residual front color placed with some margin outside the core FOV. 3. Design the lens and device opto-mechanics. Determine the expected seam width and the expected thickness and wedge variations from manufacturing and assembly tolerance differences between and within seams. Determine the expected extended FOV overlap to cover the mechanical and optical seam widths expected to be the same height as, or greater than, the expected maximum opto-mechanical wedge and thickness variations so that parallel chief rays between adjacent lens systems can be selected without underlap (for the expanded field of view). 4. Update the lens design to provide a field of view overlap along the seam between the outer compressor lens elements and to exceed both the expected maximum mechanical variations and the residual front color while controlling the parallax within it.

[0142] In conclusion, aspects of the present disclosure provide an improved panoramic multi-camera capture device having a low-parallax camera in which optomechanics enables image capture by a camera synergistically. The mechanical joints intervening between two adjacent cameras have a physical width and thus can affect the extent of the light splitting or blind regions and the value of the lost pixels or extended FOV. This application provides some means, structures, and configurations for providing robust and controlled alignment of a plurality of adjacent cameras within a panoramic device so as to reliably reduce the width of the joints.

[0143] To push the chief ray to the edge of the polygon surface, the aberrations of the entrance pupil, and in particular the spherical aberration and axial chromatic aberration of the pupil, should be optimized or reduced. In context, the entrance pupil and exit pupil, which are the projected images of the aperture stop in the object space and image space respectively, are usually a matter of academic curiosity and represent the virtual image of the aperture stop that one can see when looking into the lens.

[0144] In a typical optical system, for providing good image quality, it is typical to aim for a net small individual surface contribution, even if the values at the individual internal surfaces are on a larger scale, whether positive or negative, as the aberrations are important at the image plane. On the other hand, aberrations at the aperture stop are often not much of an issue except for positioning the stop correctly to define the lens system F-number while minimizing clipping or vignetting of the light beam's field of view. It should be noted that when the object is placed at the aperture stop, the pupil aberrations can affect the apparent image quality of the object's image as seen by a person looking at the entrance or exit pupil, although these pupil aberrations do not necessarily represent the image quality at the image plane.

[0145] In the case of a low-parallax lens, on the other hand, pupil aberration, and in particular entrance pupil aberration, is important. First, in order to properly begin designing the lens to control the parallax, the entrance pupil needs to be positioned behind the imaging plane. Second, directing the peripheral chief ray from the outer compressor lens element towards the lower parallax amount needs to be controlled. As described above, optimization of the spherical aberration of the entrance pupil can be an effective way to limit parallax error. In that context, a slight, moderately corrected, or under-corrected optimization of the spherical aberration of the entrance pupil can occur, meaning that the non-paraxial chief ray NP points can each reach or follow the paraxial NP points. Additionally, the axial chromatic aberration of the entrance pupil causes chromatic aberration that can affect the optimization of the spherical aberration of the entrance pupil. The front color, and the axial chromatic aberration itself, which can be regarded as an artifact of this axial chromatic aberration, can be reduced by the judicious selection and use of high and low dispersion optical materials within the lens design. Optimization of the spherical aberration of the entrance pupil or chief ray pointing provides fine-tuning for non-paraxial chief ray NP point pointing and parallax reduction, but the distortion from the compressor lens group also has an increasing effect on the projected chief ray pointing that goes towards a position behind the lens and towards the lower parallax amount for an increasing field of view. The magnitude and characteristics of the distortion, which also define the chief ray height on the outer surface of the outer compressor lens element, can be significantly determined by the use of aspheres within the compressor lens group.

[0146] This description has emphasized the design of the improved multi-camera image capture device 300 for use in high-bandwidth visible light, or applications perceivable by humans, but these devices can also be designed for narrow-band visible applications (modified using spectral filters, ultraviolet (UV), or infrared (IR) optical imaging applications). Polarizers or polarizer arrays can also be used. Additionally, although the imaging cameras 320 have all been described as using refractive designs, the optical design can also be reflective, or catadioptric, and combinations of refractive and reflective optical elements can be used.

Claims

1. An imaging device, A first imaging lens, comprising A first outer lens element having a polygonal shape including a plurality of first side surfaces, and A first datum feature on a first side surface among the plurality of first side surfaces The first imaging lens including the same, A second imaging lens, comprising A second outer lens element having a polygonal shape including a plurality of second side surfaces, and A second datum feature on a second side surface among the plurality of second side surfaces The second imaging lens including the same, An imaging device comprising the same, The first datum feature and the second datum feature maintain a distance between the first side surface and the second side surface, The first imaging lens images a first field of view including a first nominal field of view and a first extended field of view larger than the first nominal field of view, The second imaging lens images a second field of view including a second nominal field of view and a second extended field of view larger than the second nominal field of view, The first extended field of view and the second extended field of view overlap at a seam between the first imaging lens and the second imaging lens, A residual parallax error within the overlap of the first extended field of view and the second extended field of view is two pixels or less, An imaging device.

2. The first datum feature contacts the second datum feature. The imaging device according to claim 1.

3. The first datum feature includes one or more first convexly curved protrusions, and the second datum feature includes one or more second convexly curved protrusions. The imaging device according to claim 1.

4. The imaging device, A first housing connected to the first imaging lens, and A second housing connected to the second imaging lens, An attachment structure disposed close to the center of the imaging device, Further comprising, The first housing and the second housing are connected to the attachment structure. The imaging device according to claim 1.

5. The attachment structure includes a convex outer surface, The first housing includes a first concave surface that receives a first portion of the convex outer surface of the attachment structure, The second housing includes a second concave surface that receives a second portion of the convex outer surface of the attachment structure. The imaging device according to claim 4.

6. The attachment structure includes a plurality of concave portions formed within an outer surface, the first housing includes a first convex outer surface received within a first one of the plurality of concave portions, and the second housing includes a second convex outer surface received within a second one of the plurality of concave portions. The imaging device according to claim 4.

7. The imaging device according to claim 4, further comprising a first locking feature for holding the first housing with respect to the attachment structure and a second locking feature for holding the second housing with respect to the attachment structure.

8. The imaging device according to claim 7, wherein at least one of the first locking feature or the second locking feature includes at least one of a spring, a latch, and a cable.

9. The attachment structure includes an internal frame having a polygonal shape that nominally matches the polygonal shape of the imaging device and having a polygonal surface that nominally matches the polygonal shape of the first outer lens element and the second outer lens element. The imaging device according to claim 4.

10. The first projection of the first non-paraxial chief ray included in the incident light entering the first outer lens element converges within a first low parallax amount, The second projection of the second non-paraxial chief ray included in the incident light entering the second outer lens element converges within a second low parallax amount, The first low parallax amount and the second low parallax amount at least partially overlap, The imaging device according to claim 1.

11. The first non-paraxial point associated with the first imaging lens and the second non-paraxial point associated with the second imaging lens nominally coincide with the center of the imaging device. The imaging device according to claim 10.

12. The imaging device according to claim 1, further comprising an optical reference for at least one of tracking a first image centroid on a first imaging surface of the first imaging lens, tracking a second image centroid on a second imaging surface of the second imaging lens, or exposure correction.

13. The imaging device according to claim 1, further comprising an optically transparent dome defining an internal volume, wherein the first imaging lens and the second imaging lens are disposed within the internal volume.

14. A low parallax imaging device, A first camera configured to image a first polygonal field of view, the first camera including a first camera housing and a first outer lens having a plurality of first sides defining a first polygonal perimeter, the first camera converging a first projection of an incident non-paraxial chief ray to a first non-paraxial point within a first low parallax amount, a first camera, A second camera configured to image a second polygonal field of view, the second camera including a second camera housing and a second outer lens having a plurality of second side surfaces that define a second polygonal perimeter, one of the plurality of second side surfaces contacting one of the plurality of first side surfaces at a seam between the first camera and the second camera, the second camera including a second camera that converges a second projection of an incident non-paraxial chief ray to a second non-paraxial point within a second low parallax amount, A mounting structure proximate to the center of the low parallax imaging device, the mounting structure configured to position the first camera housing relative to the second camera housing such that the first low parallax amount at least partially overlaps the second low parallax amount, A low parallax imaging device comprising, The first outer lens images a first field of view including a first nominal field of view and a first extended field of view larger than the first nominal field of view, The second outer lens images a second field of view including a second nominal field of view and a second extended field of view larger than the second nominal field of view, The first extended field of view and the second extended field of view overlap at a seam between the first outer lens and the second outer lens, The residual parallax error within the overlap of the first extended field of view and the second extended field of view is two pixels or less, A low parallax imaging device.

15. The mounting structure includes a first surface for contacting the first camera housing and a second surface for contacting the second camera housing, the low parallax imaging device according to claim 14.

16. The mounting structure defines a hollow center, the low parallax imaging device according to claim 14.

17. The low parallax imaging device according to claim 14, wherein the first camera includes a first datum feature on one of the plurality of first side surfaces that contacts a second datum feature on one of the plurality of second side surfaces.

18. The low parallax imaging device according to claim 14, wherein the mounting structure includes a channel centering hub with a socket, and the first camera housing and the second camera housing each have a ball feature that engages with the socket.

19. The low parallax imaging device according to claim 14, wherein the mounting structure includes a ball-shaped central hub with a socket feature, and each of the first camera housing and the second camera housing has a concave socket feature that engages with the ball-shaped central hub.

20. An imaging device, A first imaging lens, A first outer lens element having a polygonal shape including a plurality of first side surfaces, and A first datum feature on a first side surface of the plurality of first side surfaces Including a first imaging lens, A second imaging lens, A second outer lens element having a polygonal shape including a plurality of second side surfaces, and A second datum feature on a second side surface of the plurality of second side surfaces Including a second imaging lens, An imaging device comprising: The first datum feature and the second datum feature maintain a distance between the first side surface and the second side surface, The imaging device, A first housing connected to the first imaging lens, A second housing connected to the second imaging lens, A mounting structure disposed close to the center of the imaging device, and Further comprising: The first housing and the second housing are connected to the mounting structure, The mounting structure includes an internal frame having a polygonal shape that nominally matches the polygonal shape of the imaging device, the internal frame having a polygonal surface that nominally matches the polygonal shape of the first outer lens element and the second outer lens element, Imaging device.

Images