Apparatus, method and computer program for use in modeling images captured by an anamorphic lens

Abstract

The present disclosure provides an apparatus, method, and computer program for modeling geometric distortion in an image captured using an anamorphic lens. The apparatus may include one or more processors and a memory that stores one or more image arrays. Each image array may store a distorted image or an undistorted image. The distorted image represents pixel values ​​of an image of a scene in a three-dimensional object space captured in a two-dimensional image space on an image plane by an imaging system having an anamorphic lens. The undistorted image represents pixel values ​​of a distortion-compensated image of a scene in a three-dimensional object space captured in a two-dimensional image space, in which information at locations in the image plane in the distorted image has been transformed to remove the geometric distortion effect of the anamorphic lens. The memory also includes instructions for configuring one or more of the processors to convert between the distorted image and the undistorted image using an anamorphic lens distortion model to map pixel values ​​at locations in the distorted image to pixel values ​​at locations in the undistorted image, the anamorphic lens distortion model having a polynomial relationship. The anamorphic lens distortion model can be used by the device to generate undistorted images in an image array from distorted images and vice versa, or to generate distorted images based on a pinhole camera model for a scene of objects in a three-dimensional object space and an anamorphic lens in the three-dimensional object space.

Description

[Technical Field]

[0001] Technical Field The present disclosure relates to the field of image processing, particularly for transforming images to account for distortions created by anamorphic lenses. [Background technology]

[0002] background Camera lenses typically create image distortions in which straight lines in a scene are rendered as slightly curved lines in the recorded image. In certain situations (e.g., architectural photography), these distortions are undesirable, and the resulting images are often post-processed using computer software to minimize the distortion. In other situations (e.g., cinematography), distortions can be aesthetically pleasing and add a distinctive character to the recorded footage.

[0003] However, even when distortions are desired, it is often important to have detailed knowledge of their exact form. Such knowledge allows one to transform a captured image to add distortions caused by the lens, e.g., to replicate them, or to remove the distortions to reveal an undistorted image, e.g., to process the image in the undistorted image plane. Enabling such transformations allows, for example, to seamlessly merge computer-generated imagery with real-world footage captured with distortion-inducing lenses. Similarly, in purely computer-generated works, it may be desirable to simulate the characteristic distortions of lenses in order to benefit from a pleasing appearance.

[0004] There is one particular class of lenses in which the aesthetic aspects of distortion are very important: cinema anamorphic lenses. Here, a combination of spherical and cylindrical optical elements is used to create a lens with different focal lengths in the vertical and horizontal axes, and the image is squeezed horizontally so that a widescreen image fits into a standard rectangular image plane. Today, these lenses are chosen almost exclusively for their unique characteristics rather than for any technical reasons.

[0005] Characterizing the complex distortion characteristics of anamorphic lenses can be difficult, and existing techniques may not be sufficient to enable accurate registration of real-world and computer-generated image components. Therefore, there is a continuing need for improved techniques for modeling and correcting lens distortion.

[0006] It is in the above context that the present disclosure has been conceived. Summary of the Invention [Means for solving the problem]

[0007] Disclosure Overview

[0006] Viewed from one aspect, the present disclosure provides an apparatus for converting between distorted and undistorted images to compensate for geometric distortions in an image caused by an anamorphic lens. The apparatus includes one or more processors and a memory that stores one or more image arrays. Each image array stores either a distorted image representing pixel values ​​of an image of a scene in three-dimensional object space captured in two-dimensional image space on an image plane by an imaging system having an anamorphic lens, or an undistorted image representing pixel values ​​of a distortion-compensated image of the scene in three-dimensional object space captured in two-dimensional image space, where information at locations in the distorted image in the image plane has been transformed to remove the geometric distortion effects of the anamorphic lens. The memory also stores instructions for configuring one or more of the processors to convert between the distorted and undistorted images using an anamorphic lens distortion model to map pixel values ​​at locations in the distorted image to pixel values ​​at locations in the undistorted image, the anamorphic lens distortion model having the following polynomial relationship:

[0008]

number

[0009] where x and y specify points in the distorted image, x' and y' specify transformed undistorted points in the undistorted image, and Dx i,j is the x-axis distortion coefficient, and Dy i,j is the y-axis distortion coefficient, and the coefficient Dx i,j and Dy i,j is non-zero for i+j=2 to characterize the decentering of the lens. The anamorphic lens distortion model can be used by a device to generate undistorted images in an image array from distorted images and vice versa, or to generate distorted images based on a pinhole camera model for an object scene in 3D object space and an anamorphic lens in 3D object space.

[0010] In the embodiment, for i+j=3, 5, 7, where i is odd and j is even, the coefficient Dx i,j is non-zero. In an embodiment, for i+j=3, 5, 7, where i is even and j is odd, the coefficient Dy i,j is non-zero. In an embodiment, Dx i,j and Dy i,j All other coefficients in are zero or null.

[0011] In an embodiment, the memory stores the location in the ray reference plane (Ref x ,Ref y ) for a location (x, y) in the distorted image.

[0012]

number

[0013] where Sx3, Sy1, and Sy3 are the anamorphic entrance pupil shift coefficients. In an embodiment, the memory further includes instructions for: transforming a scene of objects in the three-dimensional object space into an undistorted image of the scene in the image array using a pinhole camera model for the anamorphic lens in the three-dimensional object space, optionally using the determined anamorphic entrance pupil model disclosed herein above to define where light rays pass through the entrance pupil; and transforming the undistorted image of the scene in the image array into a distorted version of the image in the image array using the anamorphic lens distortion model. In an embodiment, the three-dimensional object space is a virtual object space including a computer-generated object, and the distorted version of the image of the scene in the virtual object space created using the pinhole camera model, the anamorphic lens distortion model, and optionally the anamorphic entrance pupil model is overlaid on an image of the real-world three-dimensional object space captured by the anamorphic lens.

[0014] In an embodiment, the memory further includes instructions for: transforming the distorted image of the scene in the image array into an undistorted version of the image in the image array using an anamorphic lens distortion model; and transforming the undistorted image of the scene in the image array into a projection of the image in a three-dimensional virtual object space using a pinhole camera model for the anamorphic lens in three-dimensional object space, optionally using the determined anamorphic entrance pupil model disclosed herein to define where light rays pass through the entrance pupil. In an embodiment, the distorted image of the scene is an image of real-world three-dimensional object space captured by the anamorphic lens.

[0015] In an embodiment, the memory includes a memory for receiving distorted images of the real world captured by the anamorphic lens of test grid markings having known spacing taken at different distances from the anamorphic lens along an optical axis in a real-world three-dimensional object space, determining positions of the distorted test grid markings within the image array, and calculating a distortion coefficient Dx i,j and Dy i,jTransforming the distorted grid markings into an undistorted grid of markings based on known spacing and centered on the optical axis by determining the value of the distortion coefficient Dx i,j and Dy i,j and storing the anamorphic lens distortion model stored in the memory as a value of .times. ...

[0016] In an embodiment, the memory further includes instructions for determining values ​​of anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 based on the positions (x, y) of the distorted test grid markings in the image array for the near-field test grid to determine an anamorphic entrance pupil model disclosed herein, and storing the determined values ​​as values ​​of the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 in the anamorphic entrance pupil model disclosed herein stored in the memory.

[0017] In an embodiment, the distortion coefficient Dx i,j and Dy i,j and optionally anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 are determined using an optimization algorithm to fit the anamorphic lens distortion model and optionally the anamorphic entrance pupil model to generate positions of the distorted test grid markings within the image array.

[0018] Viewed from another aspect, the present disclosure provides a method for transforming a scene of an object in three-dimensional object space to an image plane using the apparatus disclosed herein to recreate the effect of capturing the scene using an anamorphic lens. The method includes: transforming the scene of the object in three-dimensional object space to an undistorted image of the scene in an image array using a pinhole camera model for the anamorphic lens in the three-dimensional object space, optionally using the determined anamorphic entrance pupil model disclosed to define where light rays pass through the entrance pupil; and transforming the undistorted image of the scene in the image array to a distorted version of the image in the image array using the anamorphic lens distortion model.

[0019] In an embodiment, the three-dimensional object space is a virtual object space containing computer-generated objects, and a distorted version of an image of a scene in the virtual object space created using a pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model is overlaid on an image of the real-world three-dimensional object space captured by the anamorphic lens.

[0020] Viewed from another aspect, the present disclosure provides a method for converting from a distorted image of an object scene in a three-dimensional object space to a projection of the image in a three-dimensional virtual object space to counteract the effect of capturing the scene using an anamorphic lens, using an anamorphic lens distortion model. The method includes converting the distorted image of the scene in an image array to an undistorted version of the image in the image array using an anamorphic lens distortion model, and converting the undistorted image of the scene in the image array to a projection of the image in the three-dimensional virtual object space using a pinhole camera model for the anamorphic lens in the three-dimensional object space, optionally using the determined anamorphic entrance pupil model disclosed herein to define where light rays pass through the entrance pupil.

[0021] In an embodiment, the distorted image of the scene is an image of real-world three-dimensional object space captured by an anamorphic lens.

[0022] Viewed from another aspect, the present disclosure provides a computer program product bearing instructions for configuring the apparatus disclosed herein to perform the methods disclosed herein.

[0023] Viewed from another aspect, the present disclosure provides a computer-readable medium storing at least one of the determined anamorphic lens distortion model disclosed herein and the determined anamorphic entrance pupil model disclosed herein.

[0024] BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the present invention are further described below with reference to the accompanying drawings. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram of an exemplary image processing device according to aspects of the present disclosure. [Figure 2A] FIG. 1 illustrates a test grid for imaging with an anamorphic imaging system for use in characterizing the imaging system. [Figure 2B] 2B shows a distorted image of the test grid shown in FIG. 2A captured by an anamorphic imaging system, illustrating the distortion characteristics of an anamorphic lens. [Figure 2C] 2C illustrates an undistorted image of the test grid produced from the distorted image shown in FIG. 2B using the exemplary image processing device shown in FIG. 1. [Figure 3] FIG. 2 illustrates an exemplary model of an anamorphic imaging system that relates a test grid in object space to a distorted image using a pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model according to the exemplary image processing device shown in FIG. [Figure 4] FIG. 10 illustrates an exemplary anamorphic lens characterization process for determining an anamorphic entrance pupil model and an anamorphic lens distortion model for characterizing an anamorphic imaging system in which distorted images of a test grid are captured using an anamorphic lens, in accordance with aspects of the present disclosure. [Figure 5] FIG. 1 illustrates an exemplary image workflow process for converting a 3D scene in object space into a distorted image of the 3D scene that is characteristic of an anamorphic imaging system using a pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model, according to aspects of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary image workflow process for converting distorted images characteristic of anamorphic imaging systems into projections of images in 3D object space using a pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model, in accordance with aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] Detailed Description Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be recognized that the present disclosure is not limited to the embodiments, and all modifications and / or equivalents or replacements thereto also fall within the scope of the present disclosure. The same or similar reference numerals may be used throughout this specification and drawings to refer to the same or similar elements.

[0027] As used herein, the terms "have," "may have," "include," or "may include" of a feature (e.g., a number, function, operation, or component such as a part) indicate the presence of that feature and do not exclude the presence of other features.

[0028] Throughout this specification, the description and the claims, the words "comprise" and "contain" and variations thereof mean "including but not limited to" and they are not intended to (and do not) exclude other elements, integers or steps. Throughout this specification, the description and the claims, the singular includes the plural unless the context requires otherwise. In particular, where the indefinite article is used, the specification will be understood as contemplating plural as well as singular unless the context requires otherwise.

[0029] As used herein, the terms "A or B," "at least one of A and / or B," or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," or "at least one of A or B" may refer to all of: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0030] As used herein, the term "configured to" may be used interchangeably with the terms "suitable for," "capable of," "designed for," "adapted for," "made to," or "capable of," depending on the context. The term "configured to" does not inherently mean "specially designed in hardware to." Rather, the term "configured to" may mean that a device can perform an operation in conjunction with another device or component.

[0031] For example, the term "a processor configured (or set) to perform A, B, and C" may refer to a general-purpose processor (e.g., a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device, or a processor that is dedicated to performing the operations (e.g., an embedded processor).

[0032] The terms used herein are provided merely to describe some embodiments thereof and are not intended to limit the scope of other embodiments of the present disclosure. The singular forms "a," "an," and "the" are understood to include plural references unless the context clearly dictates otherwise. All terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted to have a meaning consistent with their meaning in the context of the relevant art and not in an idealized or overly formal sense unless expressly defined as such herein. In some cases, terms defined herein may be interpreted to exclude embodiments of the present disclosure.

[0033] As used throughout the figures, features or method steps are shown outlined with dashed lines to indicate that such features or method steps are optional features for provision in some embodiments, but that they may not be provided in all embodiments for implementing aspects of the present disclosure. That is, aspects of the present disclosure do not require that these optional features be included or steps be performed; they are merely included in illustrative embodiments to provide further optional implementation details.

[0034] It will be understood that any feature, integer, characteristic, or group described in connection with a particular aspect, embodiment, or example of the invention is applicable to any other aspect, embodiment, or example described herein, unless incompatible therewith. All features disclosed in this specification (including any accompanying claims, abstract, and drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The present disclosure is not limited to the details of any of the following embodiments.

[0035] Reference is now made to FIG. 1, which illustrates a schematic diagram of an exemplary image processing device 110 according to an aspect of the present disclosure.

[0036] Image processing device 110 may include an information source 111, one or more processors 112, and memory 113. Information source 111 is for providing image data or video data captured by an anamorphic imaging system characterized by device 110 to device 110 for processing. Alternatively, or in addition, information source 111 may provide computer-generated image data, video data, or 3D scene data to device 110 for processing, thereby generating image data including computer-generated data that includes distortions as if captured by the anamorphic imaging system characterized by device 110. When the image / video data and the computer-generated data are taken from different sources and processed together, a composite image may be generated by device 110 using the workflow disclosed herein, including, for example, real-world elements captured by the characterized anamorphic imaging system and computer-generated objects generated in a virtual 3D object space, where the composite image appears as if it were captured by the anamorphic imaging system. In this manner, computer-generated composite images may be generated using the image processing device 110 as if they were captured by a characterized anamorphic imaging system. The image processing device 110 may operate on data provided by the information source 111 as it is received or generated to provide real-time compositing of images as if they were captured by a characterized anamorphic imaging system. The information source 111 may generate or store data local to the image processing device 110 and may represent long-term storage such as a hard drive or solid-state drive, or in other embodiments, the information source 111 may be separate from the image processing device 110 and may generate or store data remotely and provide it to the image processing device 110 for processing. For example, the information source 111 coupled to the image processing device 110 may include an external anamorphic imaging system and / or an external virtual world environment for generating virtual objects and scenes in object space.The information from the information source 111 provided to the image processing device 110 may be in any suitable format for processing by it, including as 2D image data such as a bitmap, or as 3D scene data for imaging using a pinhole camera model of an anamorphic imaging system.

[0037] Processor 112 executes instructions that may be loaded into memory 113. Processor 112 may include any suitable number and type of processors or other devices in any suitable arrangement. Exemplary types of processor 112 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, and application specific integrated circuits.

[0038] Memory 113 may be provided by any structure capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information on a temporary or permanent basis). Memory 113 may represent random access memory or any other suitable volatile or non-volatile storage device. Memory 113 may also contain one or more components or devices that support longer-term storage of data, such as read-only memory, a hard drive, flash memory, or an optical disk, which may store software code for loading into memory 113 at runtime. In use, processor 112 and memory 113 provide a runtime environment (RTE) 114, in which instructions or code loaded into memory 113 can be executed by the processor to create instances of software modules within runtime environment 114.

[0039] Memory 113 includes instructions that, when executed by one or more processors 112, cause the one or more processors 112 to instantiate one or more image arrays 115, anamorphic lens distortion models 116, and image processing workflow modules 119. In an embodiment, memory may also include instructions that, when executed by one or more processors 112, cause the one or more processors 112 to instantiate anamorphic entrance pupil models 117 and / or lens characterization modules 120.

[0040] By implementing these constituent functional modules, apparatus 110 may be configurable by instructions stored in memory 113 and implemented in RTE 114 to perform the runtime method described in connection with Fig. 4 for determining an anamorphic entrance pupil model and an anamorphic lens distortion model to characterize an anamorphic imaging system modeled by apparatus 110. Apparatus 110 may also be configurable by instructions stored in memory 113 and implemented in RTE 114 to perform the runtime method described in connection with Fig. 5 for converting a 3D scene in object space to a distorted image of the 3D scene that is characteristic of an anamorphic imaging system. Apparatus 110 may also be configurable by instructions stored in memory 113 and implemented in RTE 114 to perform the runtime method described in connection with Fig. 6 for converting a distorted image that is characteristic of an anamorphic imaging system to a projection of the image in 3D object space.

[0041] 2A, 2B, and 2C, imaging of object space by an anamorphic imaging system and transformation of the image by image processor 110 to produce an undistorted image will now be described.

[0042] 2A shows an exemplary test grid for imaging by an anamorphic imaging system for use in characterizing the imaging system. The grid has markings at known, consistent intervals and can be imaged by the anamorphic imaging system at a distance in object space, as described below in connection with FIGS. 3 and 4, and the captured distorted image is used to determine coefficients in an anamorphic lens distortion model 116 and an anamorphic entrance pupil model 117 to characterize the anamorphic imaging system.

[0043] FIG. 2B shows a distorted image of the test grid shown in FIG. 2A captured by an anamorphic imaging system characterized by image processing device 110, illustrating the distortion characteristics of the anamorphic lens. As can be seen, due to the anamorphic lens's combination of spherical and cylindrical optical elements that provide different focal lengths in the vertical and horizontal axes, the captured image is narrowed horizontally, producing distortions characteristic of each anamorphic lens. Working with and compositing images to include computer-generated objects within this distorted image space is challenging because the composited computer-generated content may not register properly with the real-world scene, requiring extensive manual editing to synthesize an acceptable composite image that appears properly captured using the anamorphic imaging system. It should be noted that the distortion of the test grid shown in FIG. 2B due to capturing an image of the test grid of FIG. 2A using an anamorphic imaging system has been simplified and exaggerated for ease of understanding. It should be noted that accurate characterization of each anamorphic lens according to the present disclosure is necessary for proper handling and processing of images as if captured by it, for example, to enable the synthesis of computer-generated imagery or the simulation of an anamorphic imaging system using a virtual camera.

[0044] Thus, to enable the handling or creation of images that are actually or apparently captured by an anamorphic imaging system, and to facilitate seamless compositing, the image processing device 110 may be used to convert between distorted and undistorted images to add or remove the distortion effects of the anamorphic imaging system.

[0045] In this regard, image processing device 110 maintains one or more image arrays 115 in RTE 114 for receiving and storing distorted or undistorted images, such as individual images or video frames captured by an anamorphic imaging system and provided by information source 111, and / or images of a 3D virtual world scene provided by information source 111. Image processing workflow module 119 and / or lens characterization module 120 may operate on the image data stored in image array 115 to add or remove distortion effects of the anamorphic imaging system or to characterize the anamorphic imaging system.

[0046] Thus, by way of example, a distorted image of a test grid such as that shown in FIG. 2B captured by an anamorphic imaging system may be received at image array 115 of image processor 110 from information source 111 .

[0047] In use, the image processor 110 includes an anamorphic lens distortion model 116 that characterizes the anamorphic imaging system. The anamorphic lens distortion model 116 can be used to convert between distorted and undistorted images to add or remove the distortion effects of the anamorphic imaging system.

[0048] For any measured point in the distorted image stored in image array 115, anamorphic lens distortion model 116 specifies the corrections that convert the point at location (x, y) in the distorted image to a point at location (x', y') in the undistorted version of the image. The undistorted coordinates are determined by the following polynomial relationship, defined to order seven:

[0049]

number

[0050] Here, x and y specify points in a distorted image that represent pixel values ​​of an image of a scene in three-dimensional object space captured in two-dimensional image space on an image plane by an imaging system with an anamorphic lens.

[0051] Also, x' and y' specify mapped (i.e., distortion-corrected) points in the undistorted image that represent pixel values ​​of a distortion-compensated image of a scene in three-dimensional object space captured in two-dimensional image space, where information at the location in the image plane in the distorted image has been transformed to remove the geometric distortion effects of the anamorphic lens.

[0052] Dx i,j is the x-axis distortion coefficient given by equation (3), and Dy i,j is the y-axis distortion coefficient given by equation (4).

[0053]

number

[0054] As can be seen, for i+j=3, 5, 7 where i is odd and j is even, the coefficient Dx i,j is non-zero. For i+j=3, 5, 7, where i is even and j is odd, the coefficient Dy i,j is non-zero. Dx i,j and Dy i,jAll other coefficients in are zero or null.

[0055] Therefore, only 24 of the 98 distortion coefficients in Dx and Dy are non-zero. Only these specific coefficients can characterize the anamorphic imaging system to provide a high degree of accuracy while keeping the computational workload manageable. In addition, careful selection of the non-zero terms minimizes the number of degrees of freedom in the model, thereby making characterization of the manufactured lens clearer.

[0056] Importantly, as can be seen, the coefficient Dx i,j and Dy i,j are non-zero for i+j=2 to characterize the decentering of the lens. These components make it possible to adequately characterize and compensate for the distortion effects of anamorphic lenses.

[0057] By using an appropriate transformation process, such as warping, in the workflow of image processing workflow module 119 in conjunction with anamorphic lens distortion model 116, image processing device 110 can thus transform between distorted and undistorted images, and vice versa, mapping pixel values ​​between their locations to add or remove the distortion effects of the anamorphic imaging system. For example, use of anamorphic lens distortion model 116 enables image processing device 110 to transform the distorted image shown in FIG. 2B received and stored in image array 115 into an undistorted image, as shown in FIG. 2C. As can be seen, by this process, anamorphic lens distortion model 116 is used to map the pixel value of a point at location (x, y) in the distorted image shown in FIG. 2B to point (x', y') in the undistorted image shown in FIG. 2C.

[0058] The anamorphic lens distortion model 116 described above assumes that all light rays entering an anamorphic imaging system pass through a single point in 3D space, i.e., a pinhole. In this manner, the image processing workflow module 119 can use the anamorphic lens distortion model 116 in conjunction with the pinhole camera model for an anamorphic imaging system to convert between a 3D scene in object space and distorted and undistorted images of the 3D scene.

[0059] However, real lenses deviate from this pinhole camera model, and for anamorphic lenses, this can have a significant effect on reconstruction accuracy. This effect is typically important when imaging objects relatively close to the lens (approximately one meter or less).

[0060] For anamorphic lenses, these near-field distortions are particularly significant. In addition to the well-understood physical effects present in spherical lenses, anamorphic lenses are uniquely subject to distortions arising from an offset in the paraxial entrance pupil position between the vertical and horizontal axes. Thus, in an embodiment, image processor 110 includes anamorphic entrance pupil model 117, which can be used to characterize and correct the pinhole camera model for this offset, providing significantly greater accuracy in handling images of nearby objects captured using an anamorphic imaging system or computer-generated objects simulated as imaged by image processor 110 using anamorphic entrance pupil model 117 and the pinhole camera model.

[0061] 3 , which illustrates an exemplary model of an anamorphic imaging system relating a test grid in object space to a distorted image using a pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model 116, the anamorphic entrance pupil model 117 calculates the position where a ray from a location in object space O (which in this case contains the test grid of FIG. 2A ) intersects a plane called the ray reference plane P, which is perpendicular to the optical axis A and intersects the horizontal paraxial entrance pupil. The position where the ray intersects the ray reference plane P is given by the anamorphic entrance pupil model 117 and is specified by the following polynomial relationship in x and y coordinates in the captured distorted image I:

[0062]

number

[0063] Here, x and y specify the captured (i.e., distorted) point as used in the previous distortion calculation for the strictly anamorphic lens distortion model 116. Sx3, Sy1, and Sy3 are the anamorphic entrance pupil shift coefficients. (Refx, Refy, Refz) are the coordinates where the ray intersects the ray reference plane.

[0064] For the image processing workflow module 119 of the image processing device 110 to map between light rays from a scene in 3D object space O and a point on the image plane I using the pinhole camera model as fitted by the anamorphic entrance pupil model 117, only two pieces of information are needed: first, the direction of the ray through the pinhole, which is determined by its position in the image plane I or object space O, and second, the point in the ray reference plane R through which the ray passes, which is determined by the anamorphic entrance pupil model 117. In this manner, these near-field effects can be compensated for to enable accurate handling of transformations of distorted images that include near-field objects, for example, to composite computer-generated objects located close to the entrance pupil of an anamorphic imaging system in virtual object space into an image.

[0065] 3 , in an embodiment, image processing workflow module 119 of image processing device 110 implements a workflow that uses a pinhole camera model for an anamorphic lens in three-dimensional object space, adapted by anamorphic entrance pupil model 117, to define where light rays pass through the entrance pupil, in conjunction with anamorphic lens distortion model 116, to map light rays from an object scene in three-dimensional object space (such as a virtual object space received from information source 111) to a distorted image of the scene in image array 115. The workflow may accomplish this modeling and transformation from object space to distorted image in one shot, or there may be an explicit intervening step, in which the pinhole camera model adapted by anamorphic entrance pupil model 117 is used to generate an undistorted image in image array 115, before a subsequent distortion that uses anamorphic lens distortion model 116 to recreate the distortion effect of the anamorphic lens. It should be noted here that the use of the pinhole camera model, the anamorphic entrance pupil model and the anamorphic lens distortion model allows for a mapping in either direction between rays from the scene in 3D object space O and points on the image plane I.

[0066] An example process 400 implemented by the lens characterization module 120 for characterizing an anamorphic imaging system will now be described with reference to FIG. 4. Characterizing an anamorphic imaging system according to aspects of the present disclosure involves characterizing the distortion coefficients Dx of the anamorphic lens distortion model 116. i,j and Dy i,j However, if the anamorphic entrance pupil model 117 is used to fit a pinhole camera model of an anamorphic imaging system, the workflow may also include determining anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 of the anamorphic entrance pupil model 117.

[0067] Process 400 begins in step 401 with image processor 110 receiving and storing in image array 115 distorted images of the real world captured by an anamorphic lens of test grid markings having known spacing, such as those shown in FIG. 2A. As shown in FIG. 3, the images of the test grid may be taken at different distances D from the anamorphic lens along an optical axis A in the real-world three-dimensional object space O. The use of multiple distorted test grid images taken at different distances along the optical axis allows for the calculation of a distortion coefficient Dx i,j and Dy i,j and allows determination of the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 (if used) to be highly accurate with respect to position throughout object space.

[0068] In step 402, image processor 110 determines, for each distorted test grid image, the location of the distorted test grid markings in image array 115. In the example shown in FIG. 2A, this may be the intersections between horizontal and parallel lines of the test grid, the spacing between which is constant in the plane of the test grid in object space O. However, as can be seen in FIG. 2B, the spacing between these test grid intersections is distorted in the captured image by the anamorphic lens.

[0069] Therefore, in step 403, the image processor 110 calculates the distortion coefficient Dx i,j and Dy i,j , to transform the distorted grid markings into an undistorted grid of markings based on known spacing and centered on the optical axis. As explained above, the distortion coefficient Dx i,j and Dy i,j The determination of Dx allows for the generation of an anamorphic lens distortion model 116, which characterizes the distortion effects of the anamorphic imaging system. These determined values ​​are then used to calculate the distortion coefficients Dx i,j and Dy i,j is stored in the anamorphic lens distortion model 116 stored in memory as a value of

[0070] Similarly, in step 404, if an anamorphic entrance pupil model 117 is to be generated for an anamorphic imaging system, the image processor 110 determines values ​​for the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 based on the positions (x, y) of the distorted test grid markings within the image array for the near-field test grid. These determined values ​​are then stored in the anamorphic entrance pupil model 117 stored in memory as the values ​​of the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3.

[0071] In steps 403 and 404, lens characterization module 120 uses an optimization algorithm to fit the distortion coefficients Dx to the anamorphic lens distortion model and, optionally, the anamorphic entrance pupil model to generate the positions of the captured distorted test grid markings within image array 115. i,j and Dy i,jFor example, the distortion coefficients Dx that cause the anamorphic lens distortion model 116 and, optionally, the anamorphic entrance pupil model 117 to best reproduce the captured test grid image may be determined. i,j and Dy i,j and optionally a least squares fitting algorithm can be used to find values ​​for the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3.

[0072] In the embodiment shown in FIG. 1, the process 400 for characterizing the anamorphic imaging system is performed by the image processor 110 under process control by the lens characterization module 120, however, the process 400 for characterizing the anamorphic imaging system may also be performed elsewhere, for example, by the manufacturer of the anamorphic lens, and may include the distortion coefficients Dx i,j and Dy i,j It should be noted that the characterization of the anamorphic imaging system (in the form of the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3) may be received at the image processing device 110, for example, via the internet.

[0073] Once the distortion coefficient Dx i,j and Dy i,j Once the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 have been determined, the anamorphic imaging system and in particular the anamorphic lens are then characterized by the anamorphic lens distortion model 116 and optionally the anamorphic entrance pupil model 117, which can be used by the image processing workflow module 119 to add or remove the distortion effects of the characterized anamorphic lens, and to transform between a scene in 3D object space (such as one containing computer-generated content) and an image plane that directly reproduces the distortion and near-field imaging effects of the anamorphic imaging system.

[0074] Accordingly, exemplary workflow processes 500, 600 implemented by the image processing workflow module 119 to process images for a characterized anamorphic imaging system will now be described with reference to FIGS.

[0075] Referring to FIG. 5, a workflow process 500 is for transforming from a scene of objects in three-dimensional object space to an image plane to recreate the effect of capturing the scene using a characterized anamorphic lens.

[0076] In step 501, a scene of objects in a three-dimensional object space is received by image processing workflow module 119, for example from information source 111. The scene of objects in a three-dimensional object space may be of a virtual object space including computer-generated objects, or may be received in any form suitable for processing by image processing workflow module 119 for use with a pinhole camera model to produce an image of the scene in an image plane.

[0077] In step 502, image processing workflow module 119 converts the received scene of objects in the three-dimensional object space into an undistorted image of the scene in image array 115 using a pinhole camera model for the anamorphic lens in the three-dimensional object space, as illustrated in FIG. 3. In an embodiment, the anamorphic entrance pupil model 117 determined as described above may be used to fit the pinhole camera model to define where in the ray reference plane light rays pass through the entrance pupil. In this way, near-field effects of the anamorphic lens on objects in the 3D object space near the anamorphic imaging system can be taken into account in the undistorted image produced by the pinhole camera model.

[0078] Then, in step 503, the image processing workflow module 119 uses the anamorphic lens distortion model 116 to transform the undistorted images of the scene in the image array 115 into distorted versions of the images in the image array 115. The process for transforming the images then ends and may be repeated, for example, for subsequent image frames in the video.

[0079] In this manner, a distorted image of an object in 3D object space can be created that accurately replicates the capture of the object by an anamorphic imaging system. In an embodiment, a distorted version of the image of the scene in the virtual object space, created using a pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model, is overlaid on an image of the real-world 3D object space captured by the anamorphic lens. Thus, the image processing device 110 may be used to integrate computer-generated objects with real-world images captured using the anamorphic imaging system to seamlessly and accurately composite the images, without requiring manual adaptation or intervention, as if both were captured using the anamorphic imaging system. In this manner, real-time compositing of computer-generated objects within real-world footage captured by an anamorphic lens is enabled.

[0080] In the above process 500, an undistorted image is generated in the intervening step 502, but in embodiments, an undistorted image may not be generated; rather, process 500 may operate a pinhole camera model (optionally fitted with an anamorphic entrance pupil model 117) and an anamorphic lens distortion model 116 together in one step, such that light rays from 3D object space are directly mapped to positions in the distorted image of the scene stored in image array 115.

[0081] Turning now to FIG. 6, workflow process 600 is for converting from a distorted image of a scene of objects in a three-dimensional object space to a projection of the image in a three-dimensional virtual object space to counteract the effect of capturing the scene using an anamorphic lens.

[0082] In step 601, a distorted image of a scene of objects in three-dimensional object space is received by image processing workflow module 119, for example from information source 111, and stored in image array 115. The distorted image of the scene may be an image of real-world three-dimensional object space captured by an anamorphic lens.

[0083] In step 602 , the image processing workflow module 119 uses the anamorphic lens distortion model 116 to convert the distorted images of the scene in the image array 115 into undistorted versions of the images in the image array 115 .

[0084] Then, in step 603, image processing workflow module 119 transforms the undistorted image of the scene in image array 115 into a projection of the image in the three-dimensional virtual object space using a pinhole camera model for the anamorphic lens in the three-dimensional object space. The process for transforming the image then ends and may be repeated for subsequent image frames in the video, for example.

[0085] In an embodiment, the anamorphic entrance pupil model 117 determined as described above may be used to define where light rays pass through the entrance pupil to fit the pinhole camera model. In an embodiment, the distorted image of the scene is an image of real-world three-dimensional object space captured by the anamorphic lens.

[0086] In this manner, a real-world image captured by an anamorphic imaging system can be projected into a virtual object space, for example, allowing virtual objects in the virtual object space to be integrated with the scene captured by the real-world image. A distorted image of the scene in the virtual object space containing the real-world image, as if captured by the anamorphic imaging system, can then be created using process 500. In this manner, the compositing of computer-generated objects within real-world footage captured by an anamorphic lens is also enabled.

[0087] In the above process 600, an undistorted image is generated in the intervening step 602, but in embodiments, an undistorted image may not be generated; rather, process 600 may operate a pinhole camera model (optionally fitted with an anamorphic entrance pupil model 117) and an anamorphic lens distortion model 116 together in one step, such that locations in the distorted image of the scene stored in image array 115 are directly mapped to light rays in 3D object space.

[0088] 5 and 6 are merely examples, and other workflows are possible using image processor 110. For example, a captured distorted image may be transformed by image processor 110, which may make it easier to edit and add computer-generated images before subsequent distortion again to recreate the effect of an anamorphic imaging system.

Claims

1. 1. An apparatus for converting between a distorted image and an undistorted image to compensate for geometric distortions in an image caused by an anamorphic lens, comprising: one or more processors; A memory, one or more image arrays, each comprising: a distorted image representing pixel values ​​of an image of a scene in three-dimensional object space captured in two-dimensional image space on an image plane by an imaging system having an anamorphic lens; or one or more image arrays storing undistorted images representing pixel values ​​of a distortion-compensated image of the scene in the three-dimensional object space captured in two-dimensional image space, where information at locations in an image plane within the distorted image has been transformed to remove the geometric distortion effect of the anamorphic lens; instructions for configuring one or more of the processors to convert between the distorted image and the undistorted image using an anamorphic lens distortion model to map pixel values ​​at locations in the distorted image to pixel values ​​at locations in the undistorted image, the anamorphic lens distortion model having a polynomial relationship: [Equation 1] where x and y designate points in the distorted image, x' and y' designate the transformed undistorted points in the undistorted image, and Dx i,j is the x-axis distortion coefficient, and Dy i,j is the y-axis distortion coefficient, and the coefficient Dx i,j and Dy i,j is non-zero for i+j=2 to characterize the decentering of the lens, a memory, the anamorphic lens distortion model usable by the device to generate undistorted images in the image array from distorted images and vice versa, or to generate distorted images based on the scene of objects in a three-dimensional object space and a pinhole camera model for the anamorphic lens in the three-dimensional object space; An apparatus comprising:

2. For i + j = 3, 5, 7 where i is odd and j is even, the coefficient Dx i,j The apparatus of claim 1 , wherein is non-zero.

3. For i+j=3, 5, 7 where i is an even number and j is an odd number, the coefficient Dy i,j The apparatus of claim 2 , wherein is non-zero.

4. Dx i,j and Dy i,j 4. The apparatus of claim 3, wherein all other coefficients in are zero or null.

5. The memory determining a position (Refx, Refy) in a ray reference plane at the entrance pupil of the anamorphic lens through which a ray travels from the point in the three-dimensional object space relative to a position (x, y) in the distorted image using an anamorphic entrance pupil model based on the following polynomial relationship: and further comprising instructions to: [Equation 2] 2. The apparatus of claim 1, wherein Sx3, Sy1, and Sy3 are anamorphic entrance pupil shift coefficients.

6. The memory transforming a scene of objects in the three-dimensional object space into an undistorted image of the scene in the image array using a pinhole camera model for the anamorphic lens in the three-dimensional object space; transforming the undistorted images of the scene in the image array into distorted versions of the images in the image array using the anamorphic lens distortion model; The apparatus of claim 5 , further comprising instructions for:

7. 7. The apparatus of claim 6, wherein the three-dimensional object space is a virtual object space including computer-generated objects, and the distorted version of the image of the scene in the virtual object space created using the pinhole camera model, the anamorphic lens distortion model, and the anamorphic entrance pupil model is overlaid on an image of the real-world three-dimensional object space captured by the anamorphic lens.

8. The memory converting distorted images of a scene in the image array using the anamorphic lens distortion model into undistorted versions of the images in the image array; transforming the undistorted image of the scene in the image array into a projection of the image in a three-dimensional virtual object space using a pinhole camera model for the anamorphic lens in the three-dimensional object space, and using the anamorphic entrance pupil model to define where the light rays pass through the entrance pupil; The apparatus of claim 5 , further comprising instructions for:

9. The apparatus of claim 8 , wherein the distorted image of the scene is an image of a real-world three-dimensional object space captured by the anamorphic lens.

10. The memory receiving a distorted image of the real world captured by the anamorphic lens of test grid markings having known spacing taken at different distances from the anamorphic lens along an optical axis in a real-world three-dimensional object space; determining the location of the distorted test grid markings within the image array; The distortion coefficient Dx i,j and Dy i,j to transform the distorted grid markings into an undistorted grid of markings based on the known spacing and centered on the optical axis; The determined value is used to calculate the distortion coefficient Dx i,j and Dy i,j in the anamorphic lens distortion model stored in a memory as the value of The apparatus of claim 5 , further comprising instructions for:

11. The memory determining values ​​of the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 based on the position (x, y) of the distorted test grid marking within the image array for a near-field test grid to determine the anamorphic entrance pupil model; storing the determined values ​​in the anamorphic entrance pupil model stored in memory as the values ​​of the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3; The apparatus of claim 10 , further comprising instructions for:

12. The distortion coefficient Dx i,j and Dy i,j and the anamorphic entrance pupil shift coefficients Sx3, Sy1, and Sy3 are determined using an optimization algorithm to fit the anamorphic lens distortion model and the anamorphic entrance pupil model to generate the positions of the distorted test grid markings within the image array.

13. 6. A method of transforming from a scene of an object in three-dimensional object space to an image plane to recreate the effect of capturing a scene using an anamorphic lens, using the apparatus of claim 5, said method comprising: transforming a scene of objects in the three-dimensional object space into an undistorted image of the scene in the image array using a pinhole camera model for the anamorphic lens in the three-dimensional object space, and using the anamorphic entrance pupil model to define where the light rays pass through the entrance pupil; transforming the undistorted images of the scene in the image array into distorted versions of the images in the image array using the anamorphic lens distortion model; A method comprising:

14. 14. The method of claim 13, wherein the three-dimensional object space is a virtual object space including computer-generated objects, and the distorted version of the image of the scene in the virtual object space created using the pinhole camera model, an anamorphic entrance pupil model, and an anamorphic lens distortion model is overlaid on an image of the real-world three-dimensional object space captured by the anamorphic lens.

15. 10. A method of converting from a distorted image of a scene of an object in a three-dimensional object space to a projection of said image in a three-dimensional virtual object space to counteract the effect of capturing a scene using an anamorphic lens, using the apparatus of claim 5, said method comprising: converting distorted images of a scene in the image array using the anamorphic lens distortion model into undistorted versions of the images in the image array; transforming the undistorted image of the scene in the image array into a projection of the image in a three-dimensional virtual object space using a pinhole camera model for the anamorphic lens in the three-dimensional object space, and using the anamorphic entrance pupil model to define where the light rays pass through the entrance pupil; A method comprising:

16. The method of claim 15 , wherein the distorted image of the scene is an image of a real-world three-dimensional object space captured by the anamorphic lens.

17. A computer program product carrying instructions for configuring an apparatus according to claim 1 to perform the method according to claim 13.

Images