Method and system for image distortion correction for images taken using a wide-angle lens

Abstract

Computer-implemented method (1000) for camera lens distortion correction, comprising the following: Receiving (1002) position data of pixels on at least one distorted image of a scene and distorted by a wide-angle camera lens; Receiving (1004) pixel layers of an undistorted pixel grid for use in forming an undistorted image showing the scene; Determine (1006) radii extending from a distortion center and either to individual image point locations on the distorted image or to pixel locations on the undistorted pixel grid; Obtaining (1008) a ratio between distorted and undistorted radius values ​​by using at least one of the radii to look up the ratio on a lookup table, LUT; Determine (1010) a correspondence between one of the pixel positions and a corresponding image point position by using the ratio and repeating with different pixel positions or image point positions to form a plurality of correspondences; and Constructing (1012) the undistorted image by using the plurality of correspondences to place image data on the undistorted pixel layers, wherein the method further comprises dividing the image into blocks, obtaining predetermined homography coefficients to be used with a specific camera and with a specific lens on the camera, and applying homography coefficients to at least one block that are different from the homography coefficients of at least one other block.

Description

GENERAL STATE OF THE ART

[0001] Wide-angle lenses can be used in digital cameras to capture a wide panoramic image using a field of view (FOV) of 60 degrees or more for a standard wide-angle lens, and even 180 degrees or more for an ultra-wide-angle lens. The angles are centered at the camera's focal point. Such lenses are found on dedicated cameras, smartphones, digital security and surveillance (DSS) devices, vehicle-mounted cameras, and many other electronic devices with cameras. Wide-angle lenses initially provide a distorted image in which objects in the image appear to have unrealistic shapes. Objects with flat surfaces, for example, appear curved, and other objects have unusual, distorted shapes. These conventional wide-angle lens cameras have image processing units (IPUs) that correct the distortion in the images.Such camera IPUs balance several demands while correcting distortion. Distortion correction should be sufficiently precise to provide a high-quality image while also providing high performance so that images can be delivered in real time to meet video playback or transmission requirements. All this while performing such calculations with relatively low power consumption, allowing distortion correction to be performed on small, low-power devices. Conventional systems that perform distortion correction can provide suitable performance and low power consumption, even with the use of hardware-accelerated solutions, but they often fail to deliver a good-quality image. US 2005 / 0 105 822 A1 discloses an image acquisition device with variable distortion aberration. US 2016 / 0 140 713 A1 describes a system and method for modeling and calibrating an imaging device. US 2007 / 0 188 633 A1 discloses distortion correction of images using a hybrid interpolation technique. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The material described herein is illustrated in the accompanying figures by way of example and not limitation. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. For example, the dimensions of some elements relative to others may be exaggerated for clarity. Furthermore, where appropriate, reference numerals have been repeated in the different figures to identify corresponding or analogous elements. In the figures: is Fig. 1 a schematic diagram of an input image with a barrel distortion; is Fig. 2 a schematic diagram of an undistorted output image showing the image of the Fig. 1 corrected; is Fig. 3 an image of a barrel-detected input image; is Fig. 4 an image of an undistorted original image, which represents the image of the Fig. 3 corrected; is Fig. 5 is a schematic diagram of an undistorted corrected image showing the resulting pattern of pixels of a conventional distortion correction method for barrel distortion; is Fig. 6 is a schematic flowchart of a conventional lens distortion correction method; is Fig. 7 is a schematic diagram of an undistorted corrected image showing the resulting pattern of pixels of both a conventional distortion correction method and a distortion correction method according to at least one of the implementations disclosed herein for barrel distortion; is Fig. 8 is a schematic diagram of a close-up of the pixel pattern of the Fig. 7; is Fig. 9 is a graph showing the error between the pixels of a homography lens distortion correction method and a conventional bilinear interpolation lens distortion correction; is Fig. 10A is a flowchart illustrating an exemplary distortion correction method for wide-angle lens cameras using a radius ratio lookup table in accordance with at least one of the implementations disclosed herein; is Fig. 10B is a flowchart illustrating an exemplary distortion correction method for wide-angle lens cameras that implements homography and non-homography lens distortion correction in accordance with at least one of the implementations disclosed herein; is Fig. 11 is a graph illustrating spline interpolation of a distortion curve to form radius ratios and resulting from the application of at least one of the implementations described herein; is Fig. 12 is a diagram used to explain linear interpolation of radius ratios according to at least one of the implementations herein; is Fig. 13 is a graphical diagram of a non-rectangular quadrilateral of a distorted input image to be corrected into a rectangle by homography; is Fig. 14 the non-rectangular quadrilateral of a distorted input image divided into blocks; is Fig. 15 a rectangular undistorted output image grid divided into blocks; is Fig. 16 is a schematic diagram of a portion of an output image divided into blocks and showing pixel locations within a block; is Fig. Figure 17 is a schematic diagram of a portion of a distorted input image divided into blocks and showing interpolated layers within a block; is Fig. 18 is a schematic diagram of an image processing apparatus for performing the wide-angle lens distortion correction methods according to at least one of the implementations herein; are the Fig. 19A to 19C is a flowchart illustrating an exemplary detailed method of combined lens distortion correction (LDC) and homography distortion correction for wide-angle lens cameras according to at least one of the implementations disclosed herein; is Fig. 20 is a schematic diagram of an output block and an input quadrangle for explaining an example of a combined lens distortion correction (LDC) and a homography method for distortion correction according to at least one of the implementations herein; is Fig. 21 a diagram showing intermediate pixel positions corresponding to the pixel positions of the output block of the Fig. 20 and due to homography according to at least one of the implementations herein; is Fig. 22 a diagram showing a distorted and curved input quadrilateral and distorted input point positions corresponding to the intermediate positions of the Fig. 21 and are determined by using LDC according to at least one of the implementations herein; is Fig. 23 is a schematic diagram showing an input block of an input image formed with a 360-degree lens and having pixels in spherical coordinates; is Fig. 24 is a schematic diagram showing a corrected undeveloped initial block grid with initial blocks and having Cartesian coordinates according to at least one of the implementations herein; is Fig. 25 is a schematic diagram of a 360-degree input image to explain coordinate conversion according to at least one of the implementations herein; is Fig. 26 is a graph of distortion curves for barrel distortion showing the difference between the disclosed interpolation curve method and the conventional linear curve method when forming a radius ratio for image distortion correction; is Fig. Figure 27 is a graph of mean error per block for barrel distortion using a conventional distortion correction method; is Fig. 28 is a graph of mean error per block for barrel distortion using a distortion correction method according to at least one of the implementations herein; is Fig. Figure 29 is a graph of maximum error per block for barrel distortion using a conventional distortion correction method; is Fig. 30 is a graph of maximum error per block for barrel distortion using a distortion correction method according to at least one of the implementations herein; is Fig. 31 is a graph of distortion curves for mustache distortion showing the difference between the disclosed interpolation curve method and the conventional linear curve method when forming a radius ratio for image distortion correction; is Fig. 32 is a schematic diagram of an undistorted corrected image showing the resulting pattern of pixels of both a conventional distortion correction method and a distortion correction method according to at least one of the implementations disclosed herein for a corner block of mustache distortion; is Fig. 33 a schematic diagram of a close-up of the pixel pattern of the Fig. 32; is Fig. 34 is a schematic diagram of an undistorted corrected image showing the resulting pattern of pixels of both a conventional distortion correction method and a distortion correction method according to at least one of the implementations disclosed herein for a corner block of mustache distortion; is Fig. 35 a schematic diagram of a close-up of the pixel pattern of the Fig. 34; is Fig. 36 is a graph of mean error per block for mustache distortion using a conventional distortion correction method; is Fig. 37 is a graph of mean error per block for mustache distortion using a distortion correction method according to at least one of the implementations herein; is Fig. 38 is a graph of maximum error per block for mustache distortion using a conventional distortion correction method; is Fig. 39 is a graph of maximum error per block for mustache distortion using a distortion correction method according to at least one of the implementations herein; is Fig. 40 is a schematic diagram of an exemplary system for providing distortion correction for wide-angle lenses according to at least one of the implementations herein; is Fig. 41 is an illustrative diagram of an exemplary system; and illustrated Fig. 42 illustrates an exemplary device configured in accordance with at least some implementations of the present disclosure. DETAILED DESCRIPTION

[0003] One or more implementations will now be described with reference to the accompanying figures. While specific configurations and arrangements are discussed, it is understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of the description. It will be apparent to those skilled in the art that techniques and / or arrangements described herein may also be employed in a variety of systems and applications other than those described herein. The invention is defined in the independent claims. Dependent claims describe preferred embodiments.

[0004] Although the following description sets forth various implementations that may appear in architectures such as system-on-a-chip (SoC) architectures, implementations of the techniques and / or arrangements described herein are not limited to any particular architectures and / or computing systems and may be implemented by any architecture and / or computing system for similar purposes, including one or more image processing units (IPUs), each employing one or more image signal processors (ISPs).Various architectures, for example, employing multiple integrated circuit (IC) chips and / or packages, and / or employing various computing devices and / or consumer electronics (CE) devices, such as image capture devices, digital cameras, smartphones, webcams, video game panels, consoles, set-top boxes, tablet computers, laptop computers, DSS systems, vehicle vision and / or motion sensor systems, and the like, may implement the techniques and / or arrangements described herein. Although the following description may set forth numerous specific details, such as logic implementations, types and interrelationships of system components, logic partitioning / integration choices, etc., the claimed subject matter may be practiced without such specific details.In other cases, some material, such as control structures and complete software instruction sequences, may not be shown in detail so as not to obscure the material disclosed herein. The material disclosed herein may be implemented in hardware, firmware, software, or any combination thereof.

[0005] The material disclosed herein may also be implemented as instructions stored on a machine-readable medium or memory that can be read and executed by one or more processors. A machine-readable medium may include any medium and / or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.In another form, a non-transitory article, such as a non-transitory computer-readable medium, can be used with any of the above or other examples, except that it does not include a volatile signal per se. It includes these elements unlike a signal per se that can temporarily hold data in a "volatile" manner, such as RAM, etc.

[0006] References in the specification to "just one implementation," "an implementation," "an example implementation," etc., indicate that the described implementation may include a particular feature, structure, or characteristic, although not every embodiment may necessarily include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Further, where a particular feature, structure, or characteristic is described in connection with one implementation, it is further presumed that it is within the skill of those skilled in the art to practice such feature, structure, or characteristic in connection with other implementations, whether or not explicitly described herein.

[0007] Methods, apparatus, systems, and articles are described herein in connection with image distortion correction for images captured using a wide-angle lens.

[0008] As described above, some digital cameras have wide-angle lenses for capturing very wide panoramic images. Wide-angle is used here in a general sense as providing a field of view of more than 60 degrees, unless the context indicates otherwise, which is any camera lens that initially captures a distorted image due to the curvature of the lens and a relatively short focal point. Wide-angle lenses with a field of view of approximately 60 to 80 degrees may be called common or ordinary wide-angle lenses, and those over approximately 180 degrees are called ultra-wide-angle lenses, and may include cameras with fisheye lenses, 360-degree camera lenses, etc.These lenses produce images with severe geometric distortion that often need to be corrected to provide a realistic flat and / or rectangular image of a scene, so that the objects in the scene appear normal, with realistic dimensions for a user viewing the scene or an automated system that uses the locations of objects in the images to perform measurements, such as depth measurements. An image 100 (. Fig. 1), shown in a grid shape, is, for example, an image taken by a camera using a wide-angle lens and results in a distorted barrel shape, while an image 200 ( Fig. 2) shows the undistorted and desired state of the image 100 in a grid form, and which can be achieved after distortion correction of the image 100 has been performed. As can be seen in image 100, the pixel positions at intersections of the grid lines must be moved, for example, to form the undistorted image 200. An image of an undistorted input image 300 ( Fig. 3) in barrel distortion and a corrected and undistorted original image 400 ( Fig. 4), which corrects the image 300, is also shown.

[0009] The extent of distortion is typically expressed in so-called TV distortion or Standard Mobile Imaging Architecture (SMIA) TV distortion. Typical requirements for common wide-angle lens cameras are to support at least 30 to 40% SMIA TV distortion up to a 180-degree field of view (FOV) and 50% SMIA TV distortion for fisheye lenses with views greater than 180 degrees. The SMIA distortion percentage is measured by subtracting the height of the center of a distorted image (B) from the average of the heights of the left and right sides of the distorted image (A) and dividing this difference by (B): 100 (A - B) / B. For high image quality, the required rectification for height precision is in the order of 0.1 pixel position difference between the distorted position and the correct undistorted pixel position (here simply called 0.1 pixel).

[0010] With reference to Fig. 5, conventional lens distortion correction (LDC) divides a block of pixels in the image into interpolation layers using piecewise linear (PWL) interpolation. The interpolated pixel layers (or more precisely, pixels at pixel layers) are determined such that they are equally spaced from each other across all pixel rows and across all pixel columns, as in image 500, in the form of a global distortion correction map (or grid) with undistorted pixels placed at pixel locations 502 (some example pixels at the pixel locations have been darkened for clarity). Performing LDC in this manner introduces at least two errors within the block: curvature errors and homography errors.

[0011] Regarding curvature errors, if the distortion is even moderate, the conventional corrected and undistorted positions are already incorrect because they do not follow the reduced but persistent curvature of the pixel rows and columns from the distorted image that remains in the undistorted image. The actual rows and columns in the undistorted image after correction are not necessarily perfectly linear. For example, an elevation of a rectangular cardboard box may have a distorted curvature in the distorted image. This refers to color and brightness values ​​of the pixels for individual pixels that should be in pixels of a single linear row in a normal image, but are now significantly curved in the distorted wide-angle image.Ideally, the box in the corrected, undistorted image can still have some rows of color and brightness pixels that have a slight curve (and cross actual linear pixel grid lines of a screen), so that the box appears realistic and linear to the human eye viewing the undistorted image on a display screen. If conventional distortion correction forces the pixel data into equally spaced linear pixel layers instead of natural slight curves, the box may indeed appear unrealistic and of inferior quality due to, for example, the conversion function and interpolation algorithms.

[0012] With reference to Fig. 6, one reason why the curvature error occurs is due to the limitations of the distortion algorithm used to perform the piecewise linear interpolation, which causes the uniform spacing of the pixel locations. More specifically, a conventional radius-ratio-based lens distortion process 600 is provided to account for the curve errors. A distorted input image 602 may be received, and is shown here in a barrel distortion. Distortion conversion coefficients K0 and K1 may be obtained 604 and are predetermined for an individual lens (or camera). The coefficients are then used in a conversion algorithm that presents the image distortion as a radial distortion during a conversion process 606. Radial distortion is a distortion of the image along the radial direction from a distortion center 614 to the subject image 616.This forms a radial distortion R. d , as shown in the distorted image 602.

[0013] The goal of the transformation is to determine the function that can perform the mapping from the distorted image 602 to the undistorted image 608. One type is disclosed by Frederic Devernay, Olivier Faugeras, "Straight lines have to be straight: automatic calibration and removal of distortion from scenes of structured environments", Machine Vision and Applications, Springer Verlag, 2001, 13 (1), pp. 14 to 24, which proposes using a third-rank function as a mapping reference. This conventional method describes and quantifies the distortion by defining a ratio between the distorted radius R dand a corresponding undistorted radius R'' on the undistorted image 608, and extending from the distortion center 618 on the undistorted output grid (image) 608, which is the same image location as the distortion center 614 on the distorted image 602, and extending to the corresponding pixel grid location 620 corresponding to the pixel 616 on the distorted image 602. This may comprise a multi-term third-order equation such as: Rd=K0×Ru+K1×Ru3 where with a certain R d for a pixel of the distorted image R u can be calculated to assign the pixel to a pixel location on the undistorted source grid (or image). The equation can be used to calculate image coordinates that indicate where on the distorted image to fetch pixel data for a current pixel location on the undistorted image being corrected for the effect of the image distortion.

[0014] However, the desired precision cannot be achieved with the third-order multi-term function, especially for distortions higher than approximately 15% SMIA-TV. The available degrees of freedom for controlling the distortion curve are insufficient. This becomes apparent by experimenting with different types of distortions, including mustache distortion. Using higher ranks could improve this, but can result in complicated calculations for mapping an undistorted curve on the output grid to the actual distortion, and implementations are therefore more costly due to the larger amount of computation required.

[0015] With reference to the Fig. 7 and Fig. 8, for example, in an undistorted image 700, the dashed rectangle 704 represents the bounding box of an object 702 by the conventional radius-ratio-based lens distortion correction approach, and the points or pixels (and pixel locations) represent the results of the presently disclosed distortion correction LDC methodology (without homography), which are very close to the actual, correct, or ground-truth undistorted pixel locations. An error is clearly visible where the curved pixel rows of pixel locations 706 lie outside the box or object 702, as detected by the conventional distortion correction method. Fig. Figure 8 shows image 700 from close up to show this error more clearly.

[0016] These figures are provided according to small or moderate distortion, such as 10% of the SMIA-TV distortion. This relatively small distortion of only 10% SMIA-TV results in an unacceptable error of 0.8 pixels (average) and 1.2 pixels (maximum error). When such distortion is large (~50% SMIA-TV distortion), the error reaches 4.8 pixels (average) and 7.2 pixels (maximum error).

[0017] In addition to the conventional radius-ratio-based LDC approach, adding more precise homography conversion increases the precision of the resulting undistorted images when such precision is required for a number of applications (referred to above as the homography error). Some applications require precise undistorted pixel positions to provide a high-quality image or result, such as stereo rectification and digital video stabilization (DVS), which can be performed in addition to lens distortion correction. More specifically, stereo rectification is performed when a device has multiple cameras (RGB and / or infrared (IR) projector type, to name just a few examples) to measure depth.The images from the multiple cameras are rectified together, or in other words, the points on one image are mapped to points on one or more other images using algorithms such as triangulation, etc. The rectified images can then be used to form a 3D space to locate and / or measure objects within that 3D space.

[0018] DVS, which is a type of image stabilization (IS) and can be called digital image stabilization (DIS) or electronic image stabilization (EIS), typically employs feature trackers to recover image-plane (2D) motion or to extract underlying (3D) camera motion. A low-pass filter can be applied to the recovered motion, and a new video is generated by synthesizing frames along this smoothed path. DVS may also involve buffering image data near and beyond the outer boundaries of a camera frame to have image data ready in the event of camera shake that inserts regions of image data into a frame (or image) that were originally outside the frame. All of these techniques rely heavily on the precision of the undistorted, corrected image data when a wide-angle lens is used to capture the image.As mentioned, LDC alone often fails to provide a sufficiently high-quality image, so these applications can operate with a high level of precision or provide high-quality images themselves. It should be noted that the terms frame, image, and figure are used interchangeably herein.

[0019] Compared to LDC, homography provides a distinct, more precise distribution of interpolation points and can achieve a true mathematical description of the geometric transformation between two planes (such as a distorted image or frame and an undistorted image or frame). Homography is therefore a form of frame rectification. Therefore, if a camera observes a planar surface, the transformed undistorted image is further transformed using homography, and the resulting image should be precise enough to be used as another virtual viewpoint of the surface, which is often achieved with stereo cameras and DVS.

[0020] Homography conversion can be estimated in a number of ways. In one example, estimating homography may involve minimizing the sum of the squared difference in pixel grid location between the pixels on an image (the distorted image) and the candidate pixel locations, for example, on an undistorted image, and by using multiple degree-of-freedom (or direction-related matrix) coefficients or parameters, as described below.

[0021] With reference to Fig. 9, the error between conventional bilinear interpolation distortion correction and homography-based distortion correction is calculated and shown on a table 900. The errors are expressed as the distance in pixels between the required interpolation points and the current bilinear interpolation points, assuming a 32 × 128 input image block, and the table is based on an entire 512 × 512 image divided into 32 × 128 blocks. The X and Y positions of the axes are pixel locations of the undistorted image. The errors resulting from the lack of homography support range from error values ​​up to about 2 pixels, as shown. The resulting pattern shows that four 128-pixel-wide blocks have the largest error in the center block.

[0022] Regarding the hardware implementation during LDC, and for a number of computing functions, such as plug and play, real-time video viewing, and streaming to smartphones, for example, both speed and low power consumption are important. However, some conventional IPU solutions rely on the CPU for a significant amount of distortion correction computation, such as calculating undistorted radii, as described above, which increases the overall SoC power consumption, for example, when SoCs host the IPUs. In one embodiment, the performance is intended to handle 4K image resolution at a rate of 60 fps to provide real-time video, while these requirements are expected to increase dramatically in the future.When the previous solutions use image processing units (IPUs), which have a hardware block on an SoC dedicated to performing image processing and distortion correction, to achieve these goals, the performance and speed requirements can be met, but the image quality requirements are still missed.

[0023] To solve these problems, the present wide-angle lens distortion correction methods provide: (a) high precision to meet image quality requirements, (b) high performance to meet pixel throughput requirements, and (c) low power consumption, remaining within a power consumption range that small devices can support and supporting reasonable battery life, thereby enhancing the capabilities of the computing device. The present distortion correction methods precisely and in a hardware-efficient manner determine the positions of the pixels to be interpolated. The interpolation used to perform the distortion correction disclosed herein precisely follows the curvature of the actual physical distortion and takes homography into account.The distortion correction systems and methods used herein are much more efficient than conventional distortion correction methods, providing high-quality images and good performance for real-time implementations such as Plug and Play (PnP). Performance overhead is also significantly reduced by using predetermined radius ratio lookup tables for LDC instead of using the CPU, for example, to calculate undistorted radii and determine radius ratios. Combining the LDC methods and homography in a single conversion, as described below, also advantageously reduces power consumption for applications requiring the precision of homography. Finally, this method also covers ultra-wide-angle lenses, such as fisheye lenses, and even 360-degree camera lenses by adding a stage that converts spherical coordinates to Cartesian coordinates, as described below.

[0024] The present solution results in errors that are an order of magnitude smaller, both for average and maximum, than existing solutions. It has been found that, with the method disclosed herein, images with high distortion (~50% SMIA-TV distortion) result in 0.0018 pixels (average) and 0.003 pixels (maximum error), while the existing solution results in 4.8 pixels (average) and 7.2 pixels (maximum error), as mentioned above. For images with low distortion (~10% SMIA-TV distortion), the present solution results in 0.0003 pixels (average) and 0.0005 pixels (maximum error), while the existing solution results in 0.8 pixels (average) and 1.2 pixels (maximum error), as mentioned above.The present solution therefore reduces the interpolation position errors to less than 0.1 pixels (average and maximum) while taking into account both radius ratio-based lens distortion correction and homography.

[0025] With reference to Fig. 10A illustrates a process 1000 for image distortion correction for wide-angle lenses in accordance with at least some implementations of the present disclosure, and more particularly, for using a lookup table to determine radius ratios for lens distortion correction (LDC). The process 1000 may include one or more even-numbered operations 1002 through 1012. As a non-limiting example, the process 1000 may form at least a portion of an image distortion correction process for a system 1800 and / or 4000, as discussed herein and where relevant.

[0026] The process 1000 includes "receiving location data of pixels on at least one distorted image of a scene and distorted by a wide-angle camera lens" 1002. This refers to image data of a scene distorted by using a wide-angle lens, which may be a conventional wide-angle lens or ultra-wide angle, fisheye, 360 degrees, etc., as described herein. The location data may include locations of pixels in the distorted image, and the data may be preprocessed at least sufficiently to perform distortion correction. In one embodiment, the image data locations may be provided on a block-by-block basis, such as from an area covering 32×128 pixels of the distorted image.

[0027] The process 1000 includes "receiving pixel locations of an undistorted pixel grid for use in forming an undistorted image depicting the scene" 1004. The camera or system may have a preset undistorted pixel grid with known pixel locations to be assigned to pixel locations on the distorted image. According to one embodiment, these are provided in blocks such as 32 × 128 pixels.

[0028] The process 1000 includes “determining radii extending from a distortion center and to either individual pixel locations on the distorted image or pixel locations on the undistorted pixel grid” 1006. Here, the radii extend either from a distortion center on the distorted image to individual pixel locations on the distorted image (R d, as described above), or the radii extend from the distortion center projected onto the undistorted pixel grid and to pixel positions on the undistorted pixel grid (R u , as described above). Although the examples herein R u as an independent variable, R could instead d be used.

[0029] The process 1000 includes “obtaining a ratio between distorted and undistorted radius values ​​by using at least one of the radii to look up the ratio on a lookup table (LUT)” 1008. Therefore, if the LUT is set to R u the LUT returns a ratio of R d to u which can be used in an equation to determine the correct R d components, as detailed below. As mentioned, this could be the opposite of R dis input into the LUT, and the output is a ratio of R u to d In one embodiment, the input to the LUT is the square of one of the radii to avoid mathematical iterative square root operations. To improve the precision of the LUT approximation, the curves can also be treated as close to quadratic.

[0030] Using the LUT and predetermined radius ratios block by block allows for a much more efficient use of hardware, such as a dedicated graphics processing unit (GPU) or an image signal processor (ISP), since the hardware can be sized to handle calculations performed on a block-by-block basis, rather than on the entire image, to calculate the best radius ratios as performed by the conventional LDC methods.

[0031] The process 1000 includes “determining a correspondence between one of the pixel locations and a corresponding one of the image point locations by using the ratio and repeating with different pixel locations or image point locations to form a plurality of correspondences” 1010. Here, the radius ratios are applied, and the other resulting radius (be it R d or R u), can be obtained, which indicates the pixel location on the undistorted pixel grid that corresponds to a particular pixel location from the distorted image being analyzed. In one embodiment, the correspondences are obtained separately in both a horizontal and a vertical direction to provide 2D coordinates of the undistorted pixel locations by using the ratio separately in two equations (one for the x-direction, one for the y-direction). With this arrangement, a scaling value can be applied to one of the equations when distortion distances in the horizontal and vertical directions are not the same, such as when the distortion is more elliptical than circular, for example.

[0032] In one embodiment, the calculated pixel locations are applied to sample points rather than to all pixels of a single image. These sample points can be considered anchor points, and the pixels between the anchor points can be interpolated. The lookup table (LUT) can first be generated within a spline operation and can involve either the use of a parameterized model or by using images directly by determining reprojection errors, which are described in detail below.

[0033] The process 1000 includes "constructing the undistorted image by using the plurality of correspondences to place image data on the undistorted pixel layers" 1012. The undistorted image is therefore reconstructed by placing the image data of the pixels from the distorted image onto the undistorted pixel grid using the plurality of individual correspondences. Further details are provided below.

[0034] With reference to Fig. 10B, another approach is provided by a process 1050 for image distortion correction for wide-angle lenses, which is arranged in accordance with at least some implementations of the present disclosure and in particular for combining homography with LDC operations. The process 1050 may include one or more even-numbered operations 1052 through 1062. As a non-limiting example, the process 1050 may form at least a portion of an image distortion correction process for a system 1800 and / or 4000, as discussed herein and where relevant.

[0035] Process 1050 includes “receiving location data of pixels of at least one distorted image of a scene and distorted by a wide angle camera lens” 1052, and this behaves as described with respect to process 1000 and elsewhere herein.

[0036] Process 1050 includes “receiving pixel locations of an undistorted pixel grid” 1054, as described above with process 1000.

[0037] The process 1050 includes “performing a first correction operation using the pixel locations and pixel positions to generate intermediate pixel positions” 1056 and “performing a second correction operation using the intermediate pixel positions to generate modified pixel positions that form an undistorted image of the scene” 1058. The first operation therefore performs a preliminary or initial correspondence between the distorted input pixel locations and the undistorted output pixel locations, providing intermediate pixel locations (or initial undistorted locations). The second operation applies refinement to these intermediate locations to provide a more final correspondence between the pixel locations and the output pixel grid locations. These operations may or may not be consecutive, so that there may also be other intermediate refining distortion correction operations, or other operations may occur before or after these first and second operations, so that the aforementioned first and second operations are not always the only distortion correction operations and are not always consecutive.

[0038] To identify the specific type of distortion correction operations performed as the first and second operations, the process 1050 "Performing a Homography-Based Distortion Correction with Multiple Coefficients as One of the First and Second Operations" 1060 is performed. This operation comprises a two-image homography rectification. Except that instead of applying the homography to two different images from two different camera perspectives, the homography is applied to a single distorted input image and the undistorted output pixel grid. This involves the use of a 3 × 3 conversion matrix of multiple degrees of freedom coefficients, as described in detail below. The homography coefficients may be predetermined, such as during a calibration stage prior to normal consumer use and determined for each particular wide-angle lens.Since multiple coefficients can be used that specify different directions associated with an assumed epipolar line between the distorted and undistorted image, this provides a much more precise correction that better keeps the pixels along the ground truth curvature in the undistorted image.

[0039] Process 1050 may also include "performing a non-homography lens distortion correction (LDC) on radii of the pixel locations to a distortion center on the distorted image as the other of the first and second operations" 1062, and which may include one or more of the LDC operations of process 1000 as described above. As mentioned, the homography may be performed either before the LDC operations or after. It does not matter which of the two is the first and provides the initial undistorted pixel locations, as long as those locations can be refined from that of the first operation by the second operation of the other type of technical correction. In one embodiment, the LDC may be a radius ratio-based LDC, and in another embodiment, the LDC uses a radius ratio lookup table.This two-operation method can be made efficient by processing the data block by block using the LUT for the LDC, allowing the processing to be performed by a non-CPU processor. The details are outlined below. LUT-based lens distortion correction (LDC)

[0040] Now, in more detail, and as mentioned, distortion correction can be improved by using lookup tables to determine radius ratios for radial- and LUT-based LDC operations, and by combining LDC with homography to refine the mapping of pixels on the distorted input image to pixel locations on the undistorted output image. These operations, combined with the efficient use of hardware, significantly improve image quality while providing good performance and low power consumption.

[0041] The LUT-based lens distortion correction (LDC) method may include a preliminary process of generating and storing one or more lookup tables (LUTs) of radial ratios (Rd / Ru) that can be used to calculate undistorted pixel locations on an output pixel grid. These ratios are stored in the LUT not only to reduce the total amount of calculations the LDC must perform but also to eliminate mathematical division operations, further reducing the bits required to obtain the ratios for distortion correction calculations. Furthermore, the LUT may have a very limited size to achieve a hardware-efficient solution. In one example, the LUT may provide radius ratios for 32 to 512 points.In the conversions used herein, it has been found that a LUT range of 256 ratios is sufficient, and that at this level of detail, even the most extreme distortion curves are smooth. Relatively, there are no abrupt changes from one point to the next. It is understood that the range in the LUT refers to a region in image or pixel distance from a distortion center and along the radii. This represents a range of distances from the center of the distortion. The range can be in pixel or sub-pixel (fractional or sub-pel) distances, as desired. The bit depth of the input wheel radii and LUT ratios can be between 16 and 32 bits. Both the LUT range and the bit depth are modest values, resulting in efficient lower-bit conversions that can be handled by dedicated hardware, such as a GPU, ISP, etc., rather than a system CPU.

[0042] The LUT can be created during a calibration stage for a specific camera or for a Golden Camera before being made available to consumers. Alternatively, such a LUT can be transferred or otherwise delivered to consumers for initial installation or updating of the LUT on the cameras. In other options, the LUT is maintained on a server or remote computer, and the camera communicates wirelessly or wired to use the LUT to perform distortion correction. Several LUTs can be created, including one for each lens, for example, if a camera has multiple wide-angle lenses.

[0043] The LUT is first created by determining a distortion curve (R d to u) or function is generated based on a number of anchor ratio points along the curve. Then, once the function or anchor ratio points are determined, the remaining ratios for the LUT can be interpolated from them, for example, by spline interpolation, which is explained below. An exemplary way of generating the anchor ratio points for the LUT is through the use of a parameterized model, as disclosed by Zhang, Zhengyou, A Flexible New Technique for Camera Calibration, Technical Report MSR-TR-98-71, Microsoft Corp. (1998). A camera is used to take multiple images of an object with known geometry, usually a grid of squares. The parameterized model is then used to minimize the distortion errors by calculating the way an ideal camera would view the object.This may involve selecting mapping points in images and searching for geometric transformations that provide the best match between the points' locations. This parameterized model establishes the Rd / Ru ratio relationships by providing a function with a set of parameters, and then the LUT can be populated to minimize the difference, for example, by summing the absolute difference between the function defined by the parameters and the values ​​provided by the LUT, for example, through spline interpolation.

[0044] Another exemplary way to generate anchor points for the LUT is to refine the LUT by directly using the images. The optimized quantity can be a "reprojection error." See Zhang, cited above. This involves selecting assignment points in images and searching for combinations of LUT values ​​and a 3D geometric transformation that provides the best match for the points' locations.

[0045] As mentioned, an exemplary approach to generating the remaining ratios for the LUT is to use spline interpolation. The spline can be defined as: LUT(RdRu)[N]=spline(Ru,Rd,N) where R d and R uas defined above, and N is the number of ratios provided in the LUT and, in turn, the range of the LUT. To determine the ratios to be placed in the LUT, the system runs on test images by entering both the Ru values ​​and the ground-truth Rd values. This can be performed on mathematical programs such as MATLAB workbenches, which allow a user to efficiently and quickly adjust the distortion curve parameters to the real distortion using the interpolation command. One approach has been to introduce some sliders that allow control of the curve at different radial points. Once the distortion is corrected to meet certain minimum parameters, such as a maximum distortion from the true curve, the distortion curve is interpolated using spline interpolation. Here, the ground truth is obtained by setting N to an ultra-high precision of 1,000.000 and the first five input points of N to determine the ratios as follows. Input Ru:[0.00, 0.20, 0.50, 0.8, 1.00] Input Rd:[0.00, 0.35, 0.65, 0.80, 0.85] N=1,000,000

[0046] The input values ​​shown above are determined by selecting points in images with known predicted ideal positions and searching for LUT values ​​that best describe the relationship between the ideal and observed point positions.

[0047] With reference to Fig. 11, Table 1100 shows the relationship of Ru to Rd, specified in only the first five points mentioned for the LUT, and compared with simple conventional piecewise linear interpolation to demonstrate the difference in the distortion ratios calculated. The remaining points of the distortion curve dependence of Rd / Ru are calculated using ultra-high precision with 1 million points, thus defining the ground truth of the distortion in the image. As mentioned, N can be set to 256 for runtime operations.

[0048] To perform the LDC operations at runtime during normal use according to the implementations herein, the distortion correction system samples the undistorted output image (or more precisely, the undistorted pixel grid) pixel location by pixel location in a set order, such as raster scanning order. For each output pixel location (x o , y o) (where o refers to output), the undistorted radius R u , calculated. The challenge is to find a corresponding pixel position (x i , y i ), where i refers to input) on the distorted input image or pixel position grid for the distorted image, and the output pixel position (x o , y o ) being analyzed. Note that the pixel positions (x o , y o ) and (x i , y i ) image positions, measured from, for example, the top left corner of the image, as opposed to (X, Y) coordinates, which are distances from the distortion center, as explained below.

[0049] As soon as R u for individual pixel positions (x o , y o ), R u used to determine the radius ratio (R d / R u ) for this position on the LUT. The input pixel position (x i , yi ) is calculated by multiplying the ratio Rd / Ru by the respective x, y coordinate at the output pixel grid position (x o , y o ) to determine the coordinates to the distortion center as follows: Ru=Xo2+Yo2 where R u to which LUT is entered: RdRu=LUT(Ru) and where (X o , Y o ) the coordinate distances from the distortion center (X c , Y c ) and to the pixel position (x o , y o ) of (X o , Y o ). Therefore, R u and R d each have two components, and R d can have its components determined separately by applying the radius ratio separately, as with: Xi=Xo⋅RdRu Yi=Yo⋅RdRu⋅Yconst where (X i , Y i) are the distance coordinates from the distortion center on the distorted input image, which are then converted into the pixel coordinates (x i , y i ). This can be repeated n times, once for each ratio in the LUT, and where n = 0 to N - 1.

[0050] In one approach, an additional scaling parameter can be used in equations (5) or (6), or both, if the distortion is not the same in all directions from the distortion center. In particular, the distortion could be elliptical or another shape, rather than circularly concentric, and to account for this, an additional scaling factor can be multiplied by either the x-component, the y-component, or both. Here, the scaling factor is shown to be multiplied by the vertical range, and is denoted by Y in equation (6) above. const designated.

[0051] In one approach, the efficiency of implementing the above forms can be increased by removing the square root calculation. This can be achieved if the independent variable used for R d / R u -dependence is entered as a function of R u 2 instead of R u , where: Ru2=Xo2+Yo2 Rd / Ru=LUT(Ru2)

[0052] However, this approach may require more LUT capacity for larger address values ​​and, in turn, more bits per address, necessitating a deeper LUT to allow sufficient precision, such as linear interpolation between available LUT points. Alternatively, efficient square root conversion may be possible by using an iterative approach to calculating R'' or other such approaches.

[0053] In the formulas above (for example, equations (5) to (8)) it is assumed that the distortion center (X c , Y c ) has already been taken into account at (0, 0) as needed. However, if the distortion center is not at (0, 0), this can be taken into account by subtracting the coordinate of the central pixel position at the center of the distortion: (Xo−Xc)2+(Xo−Xc)2 which is typically determined during camera calibration.

[0054] With reference to Fig. 12, an exemplary way of structuring the LUT is to create an R u - or R u 2 -value for each of the N radius ratios (also called LUT points). This R u -values ​​at the LUT points may or may not be listed in the LUT. The positioning of the R'' values ​​can be omitted from the LUT if it is assumed that the R u-values ​​in a specific order and value related to N-radius ratios of the LUT points. If the actually entered R u -value does not match one of the predetermined NR u -values ​​at the LUT points, as can occur with many output pixel locations along the distortion curve between pixel locations that have an R u that corresponds to one of the radius ratio values ​​at the LUT points in the LUT, the system can calculate the radius ratio for the current R u interpolate from the radius ratios set for the N LUT points. An R ua can therefore be entered into the LUT, for example, to determine a radius ratio (R d / R u ) a to receive, while an R ua+1 can be input into the LUT, and the output is a radius ratio (R d / R u ) a+1(where a is a numerator variable, so that here a = n = 0 to N - 1). If it is desirable to have a radius ratio (and in turn an R d ) for an R u at a pixel position along the distortion curve that lies between the pixel positions of R ua and R ua+1 The interpolated radius ratio can be calculated by: RdRu=α⋅(RdRu)a+1+β⋅(RdRu)a where α is the difference between the desired R u and R ua and β is the difference between the desired R u and R ua + 1The interpolation can be performed using dedicated or other hardware, if possible other than the CPU, and using simple linear interpolation in one embodiment, but could be performed using other interpolation algorithms, such as bilinear. In one embodiment, 8 bits are used for the LUT values ​​for this linear interpolation, and the combination of the 256-range LUT and 8-bit linear interpolation has been proven to be qualitatively sufficient.

[0055] Once the radius ratios are obtained and the radii are used to find matching input distortion coordinates (X i , Y i ,) for example, using equations (5) and (6) above, and converted (x i , y i ), the pixel values ​​(for example, the chroma and luma values) of the input pixel point (x i , y i ) the output pixel positions (xo , y o ) based on the input pixel retrieved at the provided location. Since the input pixel location can be given with partial (sub-pel) precision, a sub-pel pixel location can be calculated using higher-order interpolation (such as Lanczos3 or bicubic) to obtain a smooth output image, if necessary. This is represented by: OutputImage(Xo,Yo)=Interpolate(InputImage,Xi,Yi,)

[0056] This is only one example for achieving partial precision, and the technique used can be determined independently of the distortion correction approach disclosed. Further details of these LDC operations are provided below. Homography

[0057] Homography is typically used on multiple camera stereo systems to capture multiple images from different perspectives relative to each other to form a 3D space or to create images of different perspectives of the same scene than the perspectives in the two recorded images. Here, however, homography is used in a very different way. For the purposes of homography for distortion correction, blocks of pixel locations on the distorted image can be treated as planar distorted shapes to be converted into regular shapes on the undistorted image. In one example, the distorted shapes define a quadrilateral (or 2D quadrilateral, or other shape) in the distorted image, and homography is then used to determine the output pixel grid locations (through transform matrix, linear interpolation, and other methods) to convert the quadrilateral into a rectangle on the output image.The distorted shapes form a unit or block in the distorted image. These shapes do not necessarily correspond to a planar object shown in the image, as the distortion affects wide-angle lens distortion, not the objects in the images themselves. Homography can refine the position of the interpolation points either before or after applying LDC.

[0058] With reference to the Fig. 13 to 15, a global pre-homography conversion is performed to determine homography coefficients (or parameters). The global homography is described by the set of parameters, and the parameters are used to convert the distorted shape into the normal, undistorted form, which is referred to herein as homography rectification. A selected input image plane 1302 is represented, for example, by four points labeled G0, ..., G3 on a section of an image 1300 (also referred to as a distorted plane 1402 on an image 1400 ( Fig. 14) shown), and the homography rectification transforms these points into a rectangular area 1502 on a section of an image 1500 ( Fig. 15). The units shown in the images and used below are coordinates in a range from 0 to 1, purely for the sake of simplicity in this example. An image can be divided into blocks, and each area or section can be one of these blocks for block-wise processing of these areas as desired, as explained in more detail below. Each block has different homography coefficients. Otherwise, a single homography could, for example, cover the entire image.

[0059] The global homography coefficients can be calculated using the least squares method. The homography can be viewed as: p'=Hp where p' is the distorted input image, p is the undistorted output image, and H is a 3 × 3 homography transformation matrix (also called direct homography H) with eight degrees of freedom, and can be calculated by: [wx'wy'w]=[h1h2h3h4h5h6h7h81][xy1] where w is a weight determined as w = h7x + h8y + 1, (x', y') is a pixel on the distorted image p', and (x, y) is a corresponding or rectified pixel on the undistorted pixel grid or image p, and is as described above. The homography coefficients h1 to h8 each concern a different distortion correction direction (e.g., degrees of freedom, such as translation along a specific axis, rotation around a specific axis, etc.) and can be solved using test images and the least squares method: min‖Ah−b‖2 where h is a vector of the homography coefficients of H, A is a matrix of undistorted pixel locations (x', y'), and b is a vector of corresponding distorted image points (x, y). This can be done to calculate each of the homography coefficients h1 to h8. However, the homography coefficients could be calculated in different ways by different applications, such as digital video stabilization (DVS) homography or others. A detailed explanation of an example calculation of the homography coefficients can be found in Hartley and Zisserman, Multiple View Geometry in Computer Vision, Chapter 4, Cambridge Univ. Press (2003); and Computational Photography and Introduction to Computer Vision class, Lecture 10, Homographies, Mosaics and Panoramas, MIT, (2008).

[0060] Once the homography coefficients h1 to h8 are set, these coefficients can then be progressively used to correct distortion for the particular block or blocks for the associated camera and lens in use, depending on the application. The coefficients can be modified, for example, by uploading new coefficients when the camera moves or when a new virtual view is desired in virtual vision applications. In a stereo camera application, the coefficients can be updated when the system requires recalibration. In a DVS application, the coefficients can be updated whenever the camera moves. It is also possible to divide an image into parts and have different homography conversions for each part of an image.This is typically done in DVS applications where homography is used in parts to approximate a more complex image and / or where (de)warping operations are needed for rolling shutter camera distortions.

[0061] The relationship between the coordinates of the pixels in a distorted input image x i and y i , and the coordinates in the rectified and undistorted (initial pixel grid) image with pixel positions x o and y o , can then be written as follows: xm=xi=h1xo+h2y+h3h7xo+h8yo+1 ym=yi=h4xo+h5yo+h6h7xo+h8yo+1

[0062] If the homography is performed before the LDC, as described in detail below, these resulting pixels (x i , y i ) are called interpolation points because they correspond to the LDC-interpolated input pixel positions) or intermediate points (x m , y m), where x i = x m and y i = y m , and then to form the undistorted (R u ) input to the LDC calculations, as described in detail below. 'm' is used herein to avoid confusion and can stand for modified or mean. Equations (15) and (16) can also be called the homography coefficient equations.

[0063] It is also understood that these operations can be called reverse homography, since the input to the homography coefficient equations (15) and (16) are the output pixel positions, and the result is the intermediate pixel positions (x m , y m ) or input pixel positions. The intermediate points can therefore be refined using non-homography LDC. For other alternatives, forward homography could be used instead.

[0064] With further reference to the Fig. 14-15 and now also the Fig. 16-17, in one embodiment, the homography is applied block by block. For illustration, an output image 1500 or 1600 is divided into 4 × 4 equal image blocks 1502 or 1602, respectively. The resulting input image blocks, based on the input (or intermediate) image 1700 ( Fig. 17) have intermediate points or interpolated points determined by a logical order, such as a grid order, although other orders are possible. The details of distortion correction in a block-by-block process combining homography with LUT-based LDC are provided below.

[0065] As mentioned above, although LUT-based LDC improves efficiency by reducing CPU usage due to the addition of the LUT, combining LUT-based LDC with homography provides a rectified image that enhances the quality of the distortion-corrected image. LDC can be performed before or after homography, as long as the second process further refines the results of the first process. The following example of the combined processes 1800 and 1900 provides homography before LDC.

[0066] With reference to Fig. 18, an exemplary image processing device or system 1800 is provided for implementing image distortion correction, with arrangements consistent with at least some implementations of the present disclosure, and shows alternative pipelines for implementing these operations on alternative lens types. In one embodiment, the image processing device 1800 is an imaging device or camera, or has at least one camera, and may be a smartphone with a single camera lens type, or may be a single camera with alternative lenses, or may be a device with multiple cameras, each with a different lens. In other approaches, the image processing device 1800 may not have a camera and may perform processing for the camera, such as a wireless communications server communicating over a WAN, such as the Internet.

[0067] The system 1800 shows the initial sequence of functional units in alternative pipelines for both conventional wide-angle lens images with a FOV of 60 to 180 degrees on one pipeline branch, and ultra-wide-angle, fisheye, or (360-degree) type images with a FOV of 180 to 360 degrees on other pipeline branches. The two pipelines use different operating units. In the case of conventional wide-angle lens images, a distorted input image 1802 is provided to a homography unit 1806, which performs image rectification to refine the pixel locations on the output pixel grid to form a rectangular or flat image 1808 with intermediate pixel locations. Then, an LDC unit 1810 can perform further correction of the intermediate pixel locations with respect to an undistorted output pixel grid or image 1812.This may involve the use of the LUT to provide radial relationships for performing the correction as described here.

[0068] Alternatively, in the case of ultra-wide-angle lenses, such as 360-degree images, a distorted input image 1804 may be provided to the homography unit 1806 and, as with the other wide-angle lens images, may be used to provide a flat image 1814. Only now, a dewarping unit 1816 then dewarps (or visually evolves) the image to form a flat output pixel grid or image 1818.

[0069] Of course, if the ultra-wide-angle lenses are provided in Cartesian coordinates, they could be processed using LDC instead of dewarping. Otherwise, it is understood that the order of the units shown on the System 1800 is only an example, and other orders could also be used. For example, the LDC unit and the dewarping units could perform operations before homography instead of after homography, as shown. LDC could also be applied to an image in addition to dewarping.

[0070] In one embodiment, apparatus 1800 provides a simple pipeline architecture structured to separately enable and / or disable each unit to execute the alternative pipelines. For example, dewarping is disabled when correcting common wide-angle lens distortion, while LDC is disabled for ultra-wide-angle lens image correction. This can be achieved by using a different function that maps the pixels in the output image but still uses the LUT for radial dimension conversion.

[0071] For the LDC, since the LDC has primary focus on pixel position interpolation based on the LUT, simple linear interpolation, iterative square root calculation or square diameter R u -Look up and multiply by the X o - and Y o-coordinates, the complexity added to the hardware is extremely low. Therefore, processing can be adequately provided by a GPU or ISP with a dedicated or shared geometric distortion correction (GDC) block or circuit (here referred to as the GDC hardware), which may or may not be part of an SoC.

[0072] As also mentioned above, the same dedicated GDC circuit can be used for homography, except for the calculations to generate the homography coefficients first, which can be performed by the CPU instead.

[0073] Dewarping can also be performed block by block by the GDC hardware or could be performed by the CPU.

[0074] Regarding the memory increase, the added overhead due to interpolation and LUT is the 16-bit wide or 32-bit wide LUT with 256 depth in one example and a total memory bank with 1 KB capacity.

[0075] When comparing the above-mentioned hardware elements to be added to an existing system, the additional area would be negligible. An exemplary existing system performing image processing and wide-angle distortion correction might already feature an 8 KB memory subsystem based on multiple memory banks, a register file system of approximately 150 registers, a dual-read, dual-write LUT for coefficients, each consisting of eight memory banks, each 256 bits wide and 72 bits wide, four identical MAC trees capable of generating four pixels in parallel, each using Lanczos3 interpolation techniques (6 * 6 filter footprint, 36 MAC operations per pixel, 144 total), plus additional pipeline stages and numerous FSMs controlling the entire design. Such an architecture is typically used for image processing for display images, performing compression, etc.

[0076] With reference to the Fig. 19A through 19C, an example 1900 of image distortion correction is arranged in accordance with at least one of the implementations. The process 1900 may include one or more operations 1902 through 1962, generally even-numbered. As a non-limiting example, the process 1900 may form at least a portion of an image distortion correction process for imaging device or system 1800 and 4000, as discussed herein and where relevant.

[0077] Process 1900 may include "obtaining wide-angle lens camera image(s)." The images may be in the form of raw sensor data from a camera and may be conventional wide-angle lens images (from 60 to 80 degrees FOV) or ultra-wide-angle lens images (images from 180 to 360 degrees).

[0078] Process 1900 may include "preprocessing the image at least sufficiently for distortion correction" 1904. Regardless of whether they originate from the camera or another system, the images may be processed at least sufficiently to perform distortion correction. This may involve demosaicing, noise reduction, etc. Otherwise, it could be combined with demosaicing if the interpolation task of demosaicing can be shared with the distortion correction operations. Alternatively, the methods herein could also be performed on the raw data, as long as the LUT and homography coefficients are available.

[0079] The process 1900 may include "defining blocks to be corrected in the image" 1906. Efficient GDC hardware implementation of the block can be achieved by generating the output image block by block. This allows efficient memory management and reduces the hardware area required to perform distortion correction because the hardware can be sized to process one block at a time. However, the block cannot be too small because the overall overhead for the GDC hardware calculations for each block becomes too large. In other words, each block has an overhead that initiates certain tasks, such as specific handling for edge pixels where the blocks begin and end. If the blocks are too small, there are too many overhead operations and data transfers that must be performed, which would overwhelm the system. It has been found that the smallest block should be about 32 pixels high and about 128 pixels wide.The examples herein therefore use 32 × 128 blocks.

[0080] To reduce power consumption and maintain high performance, homography and LDC should be performed in a single pass, which is achieved by using a LUT instead of iteratively calculating radius ratios. Combining these two functions or transformations into a single process can be achieved by converting them to coordinates used by both processes. This involves the following operations to provide coordinates and track the blocks through the process.

[0081] The process 1900 may include "setting the current block b = 0 of the image blocks 0 to B to be corrected" 1908. This is simply to set a block counter for the image being analyzed and is an exemplary technique, while many others are possible, depending on whether the counter can directly convey the value of the coordinates of the blocks or not.

[0082] Process 1900 may include "obtaining undistorted output pixel grid positions" 1910. The system may therefore inform the GDC hardware of the coordinates of the current output pixel grid block to be processed in order to generate the undistorted image data for the block. Knowledge of the current output position may be retained as an internal state if the output image blocks are generated in a predefined order, such as grid order, on the output grid or image. The coordinates may also be provided as input to the GDC hardware each time the GDC hardware begins processing a new output image block.

[0083] With further reference to the Fig. 16-17, the 2D position of the current output block can be called oxB and oyB, where o stands for output. For example, the upper left output block is a position oxB = 0, oyB = 0, and the coordinates increase from left to right and from top to bottom on the output image, as shown on the output image 1600. The width of the output image block is wB and the height is hB. The internal output image block coordinates are given by x. b ∈ [0, wB] and y b ∈ [0, hB], where the upper left corner of the block is a pixel or image point (x b , y b ) = (0, 0) is defined.

[0084] The output coordinates of the pixel grid position (or output image point) in the output image block at block positions oxB and oyB are calculated as: xo=xb+oxB∗wB yo=yb+oyB∗hB where the x o and y othe output pixel grid or image coordinates are calculated based on the known oxB and oyB output block position and the internal block coordinates t e , y b ) of the pixel grid point (x o , y o ) can be calculated based on the input block size. For illustration, an example output block 1604 at oxB = 2 and oyB = 1 is shown in image 1600, where block 1604 has 4 × 8 pixels (shown as an 'x'). For this operation, the GDC hardware can then be triggered to generate a block at (oxB, oyB) with the output block size of (oxdim, oydim).

[0085] The output pixel grid points x b , y b) and block layers can be set at entire pixel layers (instead of sub-pel layers) and can be any pixel layer or a uniform sample of the entire pixels. Otherwise, the output pixel grid points and / or block layers can be set at sub-pel layers instead of or in addition to the entire pixel layers. The number of pixels depends on the image resolution.

[0086] Then, homography is applied to generate a corresponding block 1704 of input pixels (also called intermediate or interpolation points) on the distorted image 1700. In particular, in one approach, homography actually works in a reverse mode, where the undistorted output pixel grid locations are obtained, and these locations are then used by applying homography to generate distorted intermediate pixel locations (x m , y m) by equations (15) and 16) described above. The calculated homography intermediate points (x m , y m ) for block 1704 are also indicated at positions each designated by an 'x' in block 1704 of image 1700.

[0087] In more detail, process 1900 may include "performing homography to change distorted pixel location" 1912. The operation of homography may initially include process 1900 performing "determining global homography mapping of distorted image into rectified output image" 1914. This refers to process 1900 operating to "determine homography coefficients" 1916.

[0088] This may refer to obtaining predetermined homography coefficients (or parameters) that have already been generated by retrieving them from memory. The homography coefficients may be generated in a prior operation using test images on the lens to obtain the homography coefficients, as described in detail above. As mentioned above, such global homography transformation that generates the homography coefficients may be computed outside the GDC hardware, for example, on the CPU, which is already present to perform other image processing tasks, such as with DVS. Otherwise, the remaining homography operations can be performed by the GDC hardware.

[0089] In some applications, such as DVS, several different homographies may exist per single frame. These could be used, for example, for complex rolling shutter distortion cases, where parts of the frame have different homographies. To support this, only the regions of a frame that are to be distortion-corrected need to be specified. These regions can then be divided into blocks, as described above. These blocks are therefore not necessarily all adjacent to each other.In these cases with multiple homographies on a single frame, each homography can be supplied to the GDC hardware either (1) in its entirety during an initialization stage, or (2) each time a block is used; the homography coefficients assigned to that block are looked up and retrieved such that each block can be initialized with a different set of parameters, if available. While the first option may involve less complex software design, as less communication may be required between the GDC hardware and a controller processor, the second option may be more suitable for hardware implementations because it requires less memory and allows uninterrupted intra-frame flow during the processing of a single block.

[0090] Process 1900 may include "Storing Homography Coefficients Locally and Accessible to Non-CPU Image Processing Hardware" 1918, and refers to storing the global homography parameters into the GDC hardware for the block being corrected. The eight parameters, each specified by a coefficient h1, ..., h8 used to define a homography conversion matrix (Equation 13 above), may be stored in one or more local onboard registers, RAM, or input buffers accessible, for example, to the GDC hardware. This may occur during an initialization stage, as mentioned above, or while fetching other corresponding block data, since the blocks are fetched one block at a time.The local input buffer of the GDC hardware can therefore be filled by the input data to be used to generate the output block located at position (oxB, oyB) on the output block grid.

[0091] With reference to Fig. 20, the process 1900 may include “Generate Local Interpixel Location” 1920. Here, the system, or GDC hardware if used, analyzes each or individual output pixel grid locations (x o , y o ) within the currently analyzed initial pixel grid block (oxB, oyB). The homography (or homography function) is applied in reverse to determine the distorted intermediate coordinates (x m , y m ) for distorted pixels using the homography coefficient equations (15) and (16) mentioned above. This stage is exemplified by an output block 200 which calculates the output pixel grid points (x o , yo ), with an overlay of the resulting distorted inscribed quadrilateral or quadrilateral 2000 corresponding to the initial pixel grid points (x o , y o ) or coincide with them. Homography is considered in the opposite direction here, because the input to the homography coefficient equations (15) and (16) are the output pixel grid points, and the output is the distorted input pixel positions. As mentioned, the system can analyze each pixel grid point or a reduced sample of the points, for example, at regular intervals.

[0092] Operations applying homography coefficient equations (15) and (16) can be made even more efficient by reducing hardware computations by using a local point calculation that utilizes the pixel grid layout and neighboring positions of the pixels. In other words, since the local block (oxB, oyB) represents a regular grid and the pixel grid positions (x o , y o ) have uniform spacing within the block, the homography coefficient equations can be implemented by simpler additions. For example, if h e x o is calculated, the next right pixel coordinate h c (x o + 1) = h e x o + h co where h co is one of the homography coefficients (or parameters) h1 to h8. Initial values ​​for each factor h e x o and h e y ocan be calculated during initialization, and each new value can be calculated by adding appropriate offsets.

[0093] Optionally, the process 1900 can have "Apply scale factor" 1922. In an example, the formulas for local calculation with a scale factor can be written as follows: s=1h7xo+h8yo+1 xm=s∗(h1xo+h2yo+h3) ym=s∗(h4xo+h5yo+h6)

[0094] This is used to avoid computationally intensive division calculations and places the denominator of the coefficient equations within the scaling factor. The calculation of the scaling factor can be implemented in hardware in several ways. One is to use a lookup table to perform the division. The lookup table has values, for example, for function 1 / D for equation (19). The denominator D of equation (19) is calculated and then used as an input to the lookup table to obtain 1 / D. This can also be done within a spline interpolation operation. An iterative implementation is another option for repeating conventional mathematical division operations, which are typically implemented in hardware, such as a machine that needs to run for a few clock cycles to achieve a certain desired precision.

[0095] With reference to Fig. 21 the result is a distorted intermediate homography quadrangle 2100 with intermediate pixel positions (x m , y m ) 2102, which correspond to interpolated points 2202 on the final distorted input block or curved iteration 2200. An image containing a number of these quadrilaterals may be called the intermediate image (for quadrilateral blocks 2100) and undistorted input images (for curved quadrilateral blocks 2200), respectively.

[0096] The process 1900 may then include “providing intermediate coordinates of distorted pixel positions” 1924. This refers either to storing the intermediate coordinates (x m , y m) and, in one example, when processing is performed block by block, and / or by providing the intermediate coordinates, for example, immediately for on-the-fly LUT-based LDC processing. This may comprise using the intermediate values ​​to calculate the input to the LUT once the intermediate values ​​are calculated and as described below. While usually the need to store intermediate values ​​is avoided by performing a complete transformation with a single upload of the pixel locations and the image point locations to local onboard storage, the system could wait for a complete block to be processed by homography before starting the LDC operations (or vice versa), in which case the intermediate coordinates may be stored in the onboard input buffer.

[0097] At this point, when the images are provided by a sensor of a conventional wide-angle lens (FOV of about 60 to 180 degrees), the process continues at A1 ( Fig. 19B) to perform lens distortion correction (LDC). Otherwise, if an image is provided from a camera with an ultra-wide lens with a FOV over 180 degrees in spherical coordinates, the process continues at A2 ( Fig. 19C) to perform dewarping instead of LDC, as with the System 1800 ( Fig. 18) explained above.

[0098] Continuing for now with process 1900, as if a conventional wide-angle lens were used to generate the images, process 1900 may include "perform lens distortion correction (LDC)" 1926, and in one embodiment, this may refer to the radius ratio-based LDC, and in other embodiments, this additionally refers to the radius ratio LUT-based LDC. The examples herein explain homography combined with radius ratio LUT-based LDC as used here for increased hardware efficiency. Alternatively, it will be appreciated that homography could be combined here with other LDC techniques that do not involve the radius ratio and LUT, which further enhance image quality due to the homography.

[0099] Continuing with the example, process 1900 may include "obtaining radii of an intermediate image and from the distortion center to intermediate pixel locations" 1928. This refers to obtaining the intermediate locations with respect to a distortion center and then using these radii as the undistorted radii as the input to the radius ratio LUT to obtain, for example, the radius ratio for the intermediate coordinate location on the distorted intermediate image 2100. The intermediate points (x m , y m ) are now used to form the undistorted radii R u used.

[0100] This operation can be performed by first projecting the distortion center to the intermediate image, such as image 2100 ( Fig. 21). The center can be shifted, for example, by equation (9), and the center (X c , Y c ) is given an arbitrary shift to corresponding intermediate image coordinates (x m-cd , ym-cd ) (where cd refers to the distortion center). This can be done once for an entire image and all blocks in the image to be corrected.

[0101] Next, X o and Y o as the x-difference and the y-difference between the (x m-cd , y m-cd ) of the distortion center and the current intermediate point (x m , y m ) is calculated. Since the differences, here intermediate distances instead of the actual initial pixel grid distances, (X o , Y o ), distances (X m , Y m ) are called.

[0102] Then equation (7) can be applied to calculate R u 2 but using intermediate distances (X m , Y m ), which are calculated here and from the intermediate image. This can, as mentioned above, be the square of R uto reduce the computational effort involved in a square root operation.

[0103] This can be repeated for each of the points in a block for which a ratio is to be provided directly from the LUT. This could be all the points in the block, as used here, some sampling, or some matching with anchor points from the final input image, which are to be used to interpolate the ratios to more points on the final input image.

[0104] The process 1900 may include "determining the radius ratio from lookup tables by using the intermediate radii" 1930. This may involve "using the square of undistorted radii as the lookup input" 1932, where undistorted may refer to the intermediate image instead. Equation (8) is therefore applied, and the R u 2value is used as the input to the radius ratio LUT to determine the ratio R d / R u for the current point, (x m , y m ) being analyzed. As described above, the radius ratio LUT can be predetermined and stored at runtime prior to distortion correction operations.

[0105] Within each block, the internal pixel position and pixel location data can be analyzed in grid-scan order. Further efficiency can be achieved from the grid-scan order of image scanning because the change in radius ratios between adjacent pixels is very small and predictable, i.e., by one pixel horizontally down a row of pixels on an image. This can be used to reduce the time consumed in performing image distortion correction by simply adding one to the 'x' component of the first R calculated in a row, and then to each x-component of an R uin the following row. Partial layers can be handled in the same way with or without varying the number added to the 'x' component. A slightly different application can be used when moving from one row to the next, but even the speed of this can be improved by remembering the value in the row above and in the same column as the current pixel layer being analyzed. For example, if the horizontal coordinate is 0 (x o = 0 or here x m = 0), the difference would be that y o (or y m ) is only increased by 1. That would make it the same as moving one pixel horizontally at a time.

[0106] The process 1900 may include “Interpolate other radius ratios” 1934. This involves determining radius ratios for those points that have an R uthat is not listed in the LUT. This can be done according to equation (10) as described above.

[0107] The process 1900 may include “determining matching pixel positions on the input image using the ratios” 1936. Here, X i and Y i , which represent the distorted distances from R d form, using equations (5) and (6), and the process 1900 may optionally include “use scale factor” 1938 to adjust for non-concentric distortion, such as Y const described above for equation (6). X i and Y i are divided into distorted input pixel positions (x i , Y i ) converted.

[0108] The results of the LDC based on the radius ratio LUT are then blocks of modified distorted input pixels (x i , y i ) 2202, as for example on the distorted input image 2200 ( Fig. 22).

[0109] The process 1900 can also have "Compensate for Tangential Distortion" 1940. In conventional correction formulas, tangential distortion is modeled by the second part of the formulas: xd=xu(1+K1r2+K2r4+⋅⋅⋅)+(P2(r2+2xu2)+2P1xuyu) yd=yu(1+K1r2+K2r4+⋅⋅⋅)+(P1(r2+2yu2)+2P2xuyu) where (x d , y d ) = (x i , y i ), (x o , Y o ) = (x u , y u ), r is the radius R u P1 and P2 are predetermined coefficients determined by experiment, and K are predetermined constants. The first part (K, K2,..., etc.) models the radial distortion and can be modeled by the proposed approximation-based radius ratio lookup table. The tangential part can be added to the LDC results according to the above formulas (22) and (23).

[0110] Since the presented GDC hardware supports a combination of homography and lens distortion, a large part of the tangential distortion can be approximated alternatively or additionally by re-adjusting the homography. This is achieved by finding homography coefficients H that minimize the effect of the tangential distortion component. For example, if P1... is known, two models are created: one with coefficients P, and another that replaces the first model with additional homography conversion. Then, the homography coefficients are adjusted to minimize the difference between the positions of a set of pixels in the image plane predicted by both models (from output coordinates to input coordinates). Other alternatives can also be considered.

[0111] The process 1900 may include “assigning pixel data of matched image points to corresponding output grid pixel locations” 1942. Therefore, pixel data such as chroma and luminance data of the matched distorted input image point (x i , y i ) the corresponding output pixel positions (x o , y o ). As mentioned above, since the input pixel location can be given with partial (sub-pel) precision, a sub-pel pixel location can be calculated using higher-order interpolation (such as Lanczos3 or bicubic) to obtain a smooth output image, if necessary, as described above with equation (11).

[0112] Process 1900 may include "b = B?" 1944. 1944. A check is now performed to determine if the last block to be corrected on the current image has been analyzed. If not, process 1900 may proceed to "get next block" 1946, and the counter is incremented by one, where process 1900 may include "set b = b + 1" 1946. The process then loops back to operation 1910 to obtain the output pixel grid locations of the next block. The process is repeated until the last block B is corrected, at which point the process ends for that image. The next image can be obtained when analyzing a sequence of images as in a video.

[0113] With reference to the Fig. 23-24, continuing with the process, for the case where an ultra-wide-angle lens is used to provide images, the process 1900 may include "dewarping pixel layers" 1950. As mentioned above, this may be applied to 360-degree lenses, but could be applied to other sizes, particularly ultra-sizes with a FOV over 180 degrees, and provided in spherical coordinates. This operation may include "determining Cartesian coordinates by using 360-degree lens parameters and dewarping block dimensions" 1952.One of the initial homography dewarping operations is applied to convert a (here modified or intermediate) block 2302 of pixels with spherical coordinates in the distorted modified input image 2300 (another example is image 1814 of system 1800) into a block 2402 of pixels with Cartesian coordinates on a flat rectangular undistorted final output image 2400 (another example is shown as image 1818 of system 1800).

[0114] To perform this conversion, the following equations effectively develop the circular image 2300 or 2500 and convert it to the original rectangular image 2400. Wd=2πr2 Hd=r2−r1 r=YoHd⋅(r2−r1)+r1 θ=xowd2π Xi=Xc+r⋅sin(θ) Yi=Yc+r⋅cos(θ) where the center of the circular image is the distortion center (X c , Y c) is assumed, but this is not always the case. r1 and r2 are the radial distances to the bottom and top of the circular image, respectively, and these are determined by lens calibration, where test images of objects with known geometry are taken, and r1 and r2 are adjusted to correctly model the geometry. The radius r to the distorted input image point (X i , Y i ) in spherical coordinates is determined by equation (26). (X o , Y o ) is the undistorted initial grid position (in units as distance to a projected distortion center) or, in the case where homography is applied first, the intermediate point position (x m , y m ).

[0115] Process 1900 may include "Use Dewarping LUT" 1954. During dewarping operations, a lookup table may also be used to implement nonlinear scaling by a vertical axis directly correlated with radius (r) in the Cartesian coordinate system. This refers to the LUT used to avoid division calculations, as mentioned above.

[0116] This results in an output image 2400 ( Fig. 24) 1818 ( Fig. 18) with dimensions Wd x Hd and undistorted output pixel grid points (X o , Y o ) (or (x m , y m )), which are distorted pixels (X i , Y i ) converted to x i , y i ) can be assigned.

[0117] The process 1900 may include "assigning pixel data of corresponding pixels to corresponding output grid pixel locations" 1956, and operates to place the chroma and luminance values ​​of the input pixels onto the assigned undistorted pixel locations, as in operation 1942 of the LDC described above.

[0118] Process 1900 may include "b = B?" 1958, and if not, process 1900 may include "Get Next Block" 1960 and "Set b = b + 1" 1962. As with the LDC process, the process loops back to operation 1910 to get the next block's data if the last block has not yet been corrected. Otherwise, the process terminates for the current frame and may be repeated for more frames.

[0119] As yet another alternative, it is possible to apply both dewarping and LDC after dewarping is completed.

[0120] It is understood that alternative processes to process 1900 are still within the scope of the disclosure even though operations of process 1900 are skipped and / or performed in a different order.

[0121] It is to be understood that each component, unit, and module described herein may include those portions of code and / or hardware that implement the operations provided by that component, unit, and module, regardless of where any of that code is located or to which additional modules, units, etc. any of that code and / or hardware is considered to belong.

[0122] It is also understood that more components may be provided on the imaging device 1800 than described here, such as encoders, decoders, transceivers, antennas, etc., many of which are mentioned in implementations 4000, 4100, and / or 4200 described below.

[0123] With reference again to the Fig. 3-4, the approach was tested on two different images, with image 300 providing a barrel distortion of ~10% SMIA-TV. Image 400 is the resulting corrected image, correcting image 300. An image providing a 50% SMIA-TV mustache distortion (not shown) was also tested. The mustache distortion was considered quite complex, as it has strong upscaling near the center and downscaling near the corners.

[0124] For each of the test images, the distortion curve is provided in comparison to the actual ground truth distortion present. The barrel distortion results and comparisons are shown on the Fig. 26, 5, 7-8 (which were already used as an example to explain distortion correction above) and 29-30. The mustache distortion results and comparisons are shown on the Fig. 31 to 39 and as follows. Barrel diagram

[0125] With reference to Fig. 26, a graph 2600 shows a distortion curve for barrel distortion, with the resulting interpolated distortion correction curve generated from the aforementioned LDC methods and shown with the image or input points on the curve. The linear curve (named simply because it is more linear) is the ground truth of the actual distortion, calculated at 1 million endpoints as described above using the MATLAB methodology. The difference shows the generated radius ratios from the disclosed methods compared to the ground truth ratios.

[0126] With further reference to the Fig. 5 and 7-8, and as already described above, image 500 shows a conventional (or legacy) barrel distortion correction that provides evenly spaced points on a grid, and images 700 and 800 compare the conventional correction of image 500 with the correction performed using the LDC methods herein (without the homography). The correct pixel positions 706 lie outside the bounding box 704 of the conventional correction.

[0127] With reference to the Fig. 27-28, a table 2700 shows the average error per block for the legacy barrel distortion correction method, whose error ranges up to 0.7, while table 2800 shows the average error per block for the barrel-distorted image using the disclosed LDC methods, whose error ranges up to about 3.5 × 10 -4pixels. Both were generated by comparing the methods with the ground truth, calculated at one million points as described above. Several orders of magnitude of difference therefore exist between the two tables, demonstrating that a significant error reduction is achieved by the disclosed methods.

[0128] With reference to the Fig. 29-30, Table 2900 shows the maximum error per block for the legacy barrel distortion correction method, whose errors range up to 1.1, while Table 3000 shows the average error per block for the barrel-distorted image using the disclosed LDC methods, whose errors range up to about 5.5 × 10 -4pixels. Both were generated by comparing the methods with the ground truth, calculated at one million points as described above. Several orders of magnitude of difference therefore exist between the two tables, demonstrating that a significant error reduction is achieved by the disclosed methods. Mustache distortion image

[0129] With reference to Fig. 31, a graph 3100 shows a distortion curve for mustache distortion, with the resulting interpolated distortion correction curve generated from the aforementioned LDC methods and shown with the image or input point on the curve. The linear curve (called simply that because it is more linear) is the ground truth of the actual distortion, calculated at 1 million endpoints as described above using the MATLAB methodology. The difference shows the generated radius ratios from the disclosed methods compared to the ground truth ratios.

[0130] With reference to the Fig. 32-33, image 3200 shows a grid 3202 of a corner block of the mustache distortion correction provided by the LDC methods disclosed herein and a bounding box 3204 providing the legacy correction, showing that correct pixel positions 3206 lie outside the bounding box 3204 of the conventional correction, demonstrating the inaccuracy of the legacy method.

[0131] With reference to the Fig. 34-35, image 3400 shows a grid 3402 of a center block of the mustache distortion correction provided by the LDC methods disclosed herein and a bounding box 3404 providing the legacy correction, showing that correct pixel positions 3406 lie outside the bounding box 3404 of the conventional correction, demonstrating the inaccuracy of the legacy method.

[0132] With reference to the Fig. 36-37, a table 3600 shows the average error per block for the legacy mustache distortion correction method, whose error ranges up to 4.0, while table 3700 shows the average error per block for the barrel-distorted image using the disclosed LDC methods, whose error ranges up to about 2.0 × 10 -3 pixels. Both were generated by comparing the methods with the ground truth, calculated at one million points as described above. Several orders of magnitude of difference therefore exist between the two tables, demonstrating that a significant error reduction is achieved by the disclosed methods.

[0133] With reference to the Fig. 38-39, a table 3800 shows the maximum error per block for the legacy mustache distortion correction method, whose errors range up to 6.0, while table 3900 shows the maximum error per block for the barrel-distorted image using the disclosed LDC methods, whose errors range up to about 3.0 × 10 -3 pixels. Both were generated by comparing the methods with the ground truth, calculated at one million points as described above. Several orders of magnitude of difference therefore exist between the two tables, demonstrating that a significant error reduction is achieved by the disclosed methods.

[0134] With reference to the Fig. 40, an exemplary system 4000 for image distortion correction for wide-angle lenses is arranged in accordance with at least some implementations of the present disclosure. In various implementations, the exemplary image processing system 4000 may include a wide-angle imaging device 4002 to form or receive captured image data. This may be implemented in various ways. Therefore, in one embodiment, the image processing system 4000 may be a digital camera or other image capture device, in which case the imaging device 4002 may be the camera hardware and the camera software, module, or device 4006.In another example, image processing system 4000 may include an imaging device 4002 that includes or is a camera, and logic modules 4004 may remotely communicate or may otherwise be communicatively coupled to imaging device 4002 to further process image data. In any of these examples, imaging device 4002 has at least one wide-angle lens 4007 of any type described above.

[0135] This technology may include a camera, such as a digital camera system, a dedicated camera device, an imaging smartphone, or an imaging tablet computer, whether a still or video camera, a device with a preview screen, or a combination thereof. Therefore, in one embodiment, the imaging device 4002 may include camera hardware and optics with one or more lenses, and one or more sensors as described above, as well as autofocus, zoom, aperture stop, ND filter, exposure compensation, flash, and actuators, and may include multiple alternative lenses. The sensor control devices may be part of a sensor module or component 4006 for operating the sensor. The control devices of the sensor component 4006 may be part of the imaging device 4002, or may be part of the logic modules 4004, or both.Such a sensor component can be used to generate images for a viewfinder, which may include a preview screen and capture still images or video. Imaging device 4002 may also include an image sensor with an RGB Bayer color filter, an analog amplifier, an A / D converter, other components for converting incident light into a digital signal, the like, and / or combinations thereof. The digital signal may also be referred to herein as raw image data.

[0136] Other embodiments include a camera sensor-type imaging device or the like (e.g., a webcam or webcam sensor or other complementary metal-oxide-semiconductor (CMOS) image sensor) without the use of a red-green-blue (RGB) depth camera and / or a microphone array to locate who is speaking. The camera sensor may also support other types of electronic shutters, such as a global shutter in addition to or instead of a rolling shutter and many other shutter types. In other examples, an RGB depth camera and / or a microphone array could be used in addition to or as an alternative to a camera sensor. In some examples, the imaging device 4002 may be provided with an eye-tracking camera.

[0137] In the illustrated example, and relevant here, logic modules 4004 may include a raw image data handling unit 4008 that performs preprocessing on the image data sufficient to correct distortion and other desired image processing functions. Logic modules 4004 may include a distortion correction unit 4010 with a homography unit 4012, an LDC unit 4014, a tangential distortion unit 4016, and a dewarping unit 4018. The logic units may also include a high-precision application unit 4020 that uses the corrected output of distortion correction unit 4010, for example, to further modify image data or display the images.

[0138] The homography unit 4012 may include a coefficient unit 4050 for generating homography coefficients and a conversion unit 4052 that applies the homography. The LDC unit 4014 may include an Ru unit 4060 for generating Rus, a ratio unit 4062 that controls the LUT to provide radius ratios, an anchor point unit 4064 that calculates Rds from the ratios, and an interpolation unit 4066 that provides interpolated ratios for pixels between the anchor points. The operation of these components is described above, and components and operations mentioned above that correspond to these components, as in the system 1800 and the process 1900, are clear from the context and names of the components and operations.

[0139] The components of the logic modules 4004 comprising these components of the distortion correction unit 4010 may be operated by, or even located entirely or partially on, at least one processor 4030, which may include one or more image processing units (IPUs) 4048, each with or access to its own one or more GDC hardware image signal processors (ISPs) and / or graphics processing units (GPUs) 4036, and which may include other hardware manipulation functionality. These IPU units 4048 may or may not be exclusively dedicated to performing distortion correction tasks, as described above.

[0140] The logic modules 4004 may be communicatively connected to the components of the imaging device 4002 to receive raw image data. Optionally, data may be provided from a data stream separate from the raw image data stream. In these cases, the logic modules 4004 are assumed to be separate from the imaging device. This need not be the case, and the logic modules may very well be considered part of the imaging device.

[0141] System 4000 may include one or more processors 4030 and memory storage 4034. In the example of system 4000, memory storage 4034 may store image content, such as captured images (or image frames) and data related to distortion correction, including any of the lookup tables 4038 mentioned above. This may also include any input, intermediate computation, or output data related to distortion correction, or other data as discussed herein.

[0142] The processor(s) 4030 may also include onboard memory, such as one or more buffers, to maintain GDC data being processed, such as intermediate pixel coordinates, ratios, or data selected from one of the LUTs, etc.

[0143] The ISP and / or GPU 4036 may include any number and type of graphics processing units that can provide the operations discussed herein. Such operations may be implemented via software or hardware, or a combination thereof. For example, the image processing unit 4036 may include circuitry dedicated to manipulating images obtained from the memory stores 4034. The processor(s) 4030 may include any number and type of processing units or modules that can provide control and other higher-level functions for the system 4000 and / or can provide any operations discussed herein. Memory stores 4034 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.).) or non-volatile memory (e.g., flash memory, etc.), and so on. In one non-limiting example, the memory storages 4034 may be implemented by a cache. Otherwise, the memory 4034 may be an EEPROM or device data system. In one embodiment, one or more units of the image processing system 4000 and the distortion correction unit 4010 may be implemented via an execution unit (EU) of ISPs 4036 on an IPU 4048. The EU may, for example, include programmable logic or circuitry, such as a logic core or logic cores that can provide a wide range of programmable logic functions. In one embodiment, any of these units may be implemented via dedicated hardware, such as fixed-function circuitry or the like.Fixed function circuitry may include dedicated logic or circuitry and may provide a set of fixed function access points that may be mapped to the dedicated logic for a fixed purpose or function.

[0144] Various components of the system described herein may be implemented in software, firmware, and / or hardware, and / or any combination thereof. For example, various components of the imaging device or system 1800 or 4000 may be provided at least in part by hardware of a computing system-on-chip (SoC), such as may be found in a computing system such as a smartphone. Those skilled in the art will recognize that systems described herein may include additional components not depicted in the corresponding figures. For example, the systems discussed herein may include additional components, such as communication modules and the like, which are not depicted for clarity.

[0145] Although an implementation of the example processes discussed herein may include performing all of the operations shown in the order illustrated, the present disclosure is not limited in this regard, and an implementation of the example processes herein may, in various examples, include only a subset of the operations shown, operations performed in a different order than the illustrated order, or additional operations.

[0146] Additionally, any one or more of the operations discussed herein may be performed in response to instructions provided by one or more computer program products. Such program products may include signal-bearing media providing instructions that, when executed by, for example, a processor, may provide the functionality described herein. The computer program products may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more graphics processing units or one or more processor cores may perform one or more of the operations of the example processes herein in response to program code and / or instructions or instruction sets communicated to the processor through one or more machine-readable media.In general, a machine-readable medium may carry software in the form of program code and / or instructions or sets of instructions that may cause any of the devices and / or systems described herein to implement at least portions of the imaging device or system 1800 or 4000 or systems or any other module or device as discussed herein.

[0147] As used in any implementation described herein, the term "module" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. The software may be embodied as a software package, code, and / or instruction set or instructions, and "hardware," as used herein in any implementation, may include, for example, individually or in any combination, wired circuitry, programmable circuitry, state machine circuitry, fixed-function circuitry, execution unit circuitry, and / or firmware storing instructions executed by programmable circuitry.The modules can be implemented collectively or individually as a circuit arrangement that forms part of a larger system, such as an integrated circuit (IC), a system-on-chip (SoC), and so on.

[0148] As used in any implementation described herein, the term "module" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. The logic units may be embodied collectively or individually as circuitry that forms part of a larger system, such as an integrated circuit (IC), a system-on-chip (SoC), and so forth. A logic unit may, for example, be embodied in logic circuitry implementing firmware or hardware of the coding system discussed herein.One of ordinary skill in the art will understand that the operations performed by hardware and / or firmware may alternatively be performed via software, which may be embodied as a software package, code, and / or an instruction set or instructions, and will also understand that the logic unit may also employ a portion of software to implement its functionality.

[0149] As used in any implementation described herein, the term "device" may refer to a module or logic unit, as those terms are described above. Accordingly, the term "device" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. For example, one of ordinary skill in the art will understand that the operations performed by hardware and / or firmware may alternatively be performed via a software module, which may be embodied as a software package, code, and / or an instruction set, and will also understand that the logic unit may also employ a portion of software to implement its functionality.

[0150] Fig. 41 is an illustrative diagram of an example system 4100 arranged in accordance with at least some implementations of the present disclosure. In various implementations, system 4100 may be a media system, although system 4100 is not limited to this context. For example, system 4100 may be incorporated into a personal computer (PC), a laptop computer, an ultra-laptop computer, a tablet, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a cellular phone, a combined cellular phone / PDA, a television, a smart device (e.g., a smartphone, a smart tablet, or a smart TV), a mobile internet device (MID), a messaging device, a data communication device, cameras (e.g., point-and-shoot cameras, super-zoom cameras, digital single-lens reflex (DSLR) cameras), and so on.

[0151] In various implementations, system 4100 includes a platform 4102 coupled to a display 4120. Platform 4102 may receive content from a content device, such as one or more content service devices 4130 or one or more content delivery devices 4140, or other similar content sources. A navigation control device 4150, including one or more navigation features, may be used to interact with, for example, platform 4102 and / or display 4120. Each of these components is discussed in more detail below.

[0152] In various implementations, the platform 4102 may include any combination of a chipset 4105, a processor 4110, a memory 4112, an antenna 4113, a storage 4114, a graphics subsystem 4115, applications 4116, and / or radio 4118. The chipset 4105 may provide intercommunication between the processor 4110, the memory 4112, the storage 4114, the graphics subsystem 4115, the applications 4116, and / or the radio 4118. For example, the chipset 4105 may include a memory adapter (not shown) capable of providing intercommunication with the memory 4114.

[0153] Processor 4110 may be implemented as a complex instruction set computer (CISC) or reduced instruction set computer (RISC) processor, x86 instruction set compatible processor, multi-core or any other microprocessor or central processing unit (CPU). In various implementations, processor 4110 may be one or more dual-core processors, one or more dual-core mobile processors, and so on.

[0154] The memory 4112 may be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

[0155] Memory 4114 may be implemented as a non-volatile storage device, such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, battery-backed SDRAM (synchronous DRAM), and / or a network-accessible storage device. In various implementations, memory 4114 may include technology to increase enhanced storage performance protection for valuable digital media, for example, when including multiple hard disk drives.

[0156] The graphics subsystem 4115 may perform processing of images, such as still or video images, for display. The graphics subsystem 4115 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple the graphics subsystem 4115 and the display 4120. The interface may be, for example, one of the following: a High-Definition Multimedia Interface (HDMI), a display port, wireless HDMI, and / or wireless HD compliant technologies. The graphics subsystem 4115 may be integrated with the processor 4110 or the chipset 4105. In some implementations, the graphics subsystem 4115 may be a standalone device communicatively coupled to the chipset 4105.

[0157] The graphics and / or video processing techniques described herein may be implemented in various hardware architectures. For example, graphics and / or video functionality may be integrated into a chipset. Alternatively, a discrete graphics and / or video processor may be used. As yet another implementation, the graphics and / or video functions may be provided by a general-purpose processor including a multi-core processor. In further embodiments, the functions may be implemented in a consumer electronics device.

[0158] The radio 4118 may include one or more radio devices capable of transmitting and receiving signals using various suitable wireless communication technologies. Such technologies may include communications across one or more wireless networks. Example wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular mobile networks, and satellite networks. When communicating across such networks, the radio 4118 may operate in accordance with one or more applicable standards in any version.

[0159] In various implementations, display 4120 may include any type of television-like monitor or television-like display. For example, display 4120 may include a computer display screen, a touchscreen display, a video monitor, a television-like device, and / or a television. Display 4120 may be digital and / or analog. In various implementations, display 4120 may be a holographic display. Also, display 4120 may be a transparent surface capable of receiving a visual projection. Such projections may convey various forms of information, images, and / or objects. For example, such projections may be visual overlays for a mobile augmented reality (MAR) application. Under the control of one or more software applications 4116, platform 4102 may display a user interface 4122 on display 4120.

[0160] In various implementations, the content service device(s) 4130 may be hosted by any national, international, and / or independent service and thus are accessible to the platform 4102 via the Internet, for example. The content service device(s) 4130 may be coupled to the platform 4102 and / or the display 4120. The platform 4102 and / or the content service device(s) 4130 may be coupled to a network 4160 for communicating (e.g., sending and / or receiving) media information to and from a network 4160. The content delivery device(s) 4140 may also be coupled to the platform 4102 and / or the display 4120.

[0161] In various implementations, the content service device(s) 4130 may include a cable television box, a personal computer, a network, a telephone, internet-enabled devices, or a device capable of transmitting digital information and / or digital content, and any other similar device capable of unidirectionally or bidirectionally communicating content between content providers and the platform 4102 and display 4120 via the network 4160 or directly. It is understood that the content may be communicated unidirectionally and / or bidirectionally to and from any of the components in the system 4100 and a content provider via the network 4160. Examples of content may include any media information, including, for example, video, music, medical and gaming information, and so on.

[0162] The content service device(s) 4130 may receive content such as cable television programs that include media information, digital information, and / or other content. Examples of content providers may include any provider of television, radio, or internet content via cable or satellite. The examples provided are not intended to limit implementations according to the present disclosure in any way.

[0163] In various implementations, platform 4102 may receive control signals from a navigation controller 4150 having one or more navigation features. For example, the navigation features of controller 4150 may be used to interact with user interface 4122. In various embodiments, navigation controller 4150 may be a pointing device, which may be a computer hardware component (particularly, a user interface device) that allows a user to input spatial (e.g., continuous or multidimensional) data into a computer. Many systems, such as graphical user interfaces (GUIs) and televisions and monitors, allow the user to control and provide data to the computer using physical gestures.

[0164] Movements of the navigation features of controller 4150 may be replicated on a display (e.g., display 4120) by movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display. For example, under the control of software applications 4116, the navigation features located on navigation controller 4150 may be mapped to virtual navigation features displayed, for example, on user interface 4122. In various embodiments, controller 4150 may not be a separate component, but may be integrated into platform 4102 and / or display 4120. However, the present disclosure is not limited to the elements or context illustrated or described herein.

[0165] In various implementations, drivers (not shown) may include technology that allows users to instantly turn the platform 4102 on and off like a television with the touch of a button after initial boot, if enabled. Program logic may allow the platform 4102 to stream content to the media adapters or other content service device(s) 4130 or content delivery device(s) 4140 even when the platform is turned "off." Additionally, the chipset 4105 may include, for example, hardware and / or software support for 5.1 surround sound audio and / or high-resolution 7.1 surround sound audio. Drivers may include a graphics driver for integrated graphics platforms. In various embodiments, the graphics driver may include a Peripheral Component Interconnect (PCI) Express graphics card.

[0166] In various implementations, any one or more of the components shown in system 4100 may be integrated. For example, platform 4102 and content service device(s) 4130 may be integrated, or platform 4102 and content delivery device(s) 4140 may be integrated, or platform 4102, content service device(s) 4130, and content delivery device(s) 4140 may be integrated, for example. In various embodiments, platform 4102 and display 4120 may be an integrated unit. For example, display 4120 and content service device(s) 4130 may be integrated, or display 4120 and content delivery device(s) 4140 may be integrated. These examples are not intended to limit the present disclosure.

[0167] In various embodiments, system 4100 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 4100 may include components and interfaces capable of communicating over a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, and so on. An example of wireless shared media may include portions of a wireless spectrum, such as the RF spectrum, and so on.When implemented as a wired system, the system 4100 may include components and interfaces capable of communicating over wired communication media, such as input / output (I / O) adapters, physical connectors for connecting the I / O adapter to a corresponding wired communication medium, a network interface card (NIC), a disk controller, a video controller, an audio controller, and the like. Examples of wired communication media may include a wire, a cable, metal lines, a printed circuit board (PCB), backplanes, switch fabrics, semiconductor material, a twisted pair, a coaxial cable, optical fibers, and so on.

[0168] Platform 4102 may create one or more logical or physical channels for communicating information. The information may include media information and control information. Media information may refer to any data representing content intended for a user. Examples of content may include, for example, data from a voice conversation, a video conference, a streaming video, an electronic mail ("email") message, a voicemail message, alphanumeric symbols, graphics, images, video, text, and so on. Data from a voice conversation may include, for example, voice information, pause periods, background noise, comfort sounds, tones, and so on. Control information may refer to any data representing commands, instructions, or control words intended for an automated system.For example, control information can be used to route media information through a system or to instruct a node to process the media information in a predetermined manner. However, implementations are not limited to the elements or context described in . Fig. 41 are shown or described.

[0169] With reference to Fig. 42, a small form factor device 4200 is an example of the varying physical configurations or form factors in which systems 4000 or 4100 may be embodied. Through this approach, the device 4200 may be implemented as a mobile computing device having wireless capabilities. A mobile computing device may refer to any device with a processing system and a mobile power source or supply, such as one or more batteries.

[0170] As described above, examples of a mobile computing device may include a personal computer (PC), a laptop computer, an ultra-laptop computer, a tablet, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a mobile phone, a mobile phone / PDA combination, a television, a smart device (e.g., a smartphone, a smart tablet, or a mobile smart TV), a mobile internet device (MID), a messaging device, a data communication device, cameras (e.g., compact cameras, superzoom cameras, digital single-lens reflex (DSLR) cameras), and so on.

[0171] Examples of a mobile computing device may also include computers configured to be worn by a person, such as wrist computers, finger computers, ring computers, eyeglass computers, belt buckle computers, wristband computers, shoe computers, clothing computers, and other wearable computers. In various implementations, a mobile computing device may be implemented, for example, as a smartphone capable of executing computer applications as well as voice communications and / or data communications. Although some embodiments may be described by way of example with a mobile computing device implemented as a smartphone, it is understood that other implementations may be implemented using other wireless mobile computing devices. The implementations are not limited in this context.

[0172] As in Fig.42, device 4200 may include a housing with a front side 4201 and a back side 4202. Device 4200 includes a display 4204, an input / output (I / O) device 4206, and an integrated antenna 4208. Device 4200 may also include navigation features 4212. I / O device 4206 may include any suitable I / O device for entering information into a mobile computing device. Examples of I / O device 4206 may include an alphanumeric keyboard, a numeric keypad, a touchpad, input keys, buttons, switches, microphones, speakers, a speech recognition device, and software, among others. Information may also be entered into device 4200 using a microphone 4214 or may be digitized by a speech recognition device.As shown, device 4200 may include a camera 4205 (e.g., including a lens, an aperture, and an image sensor) and a flashlight 4210 integrated into the back 4202 (or elsewhere) of device 4200. Implementations are not limited in this context.

[0173] Various forms of devices and processes described herein may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., resistors, capacitors, inductors, and so on), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), a field-programmable gate array (FPGA), logic gates, registers, a semiconductor device, chips, microchips, chipsets, and so on.Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (APIs), instruction sets, data processing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an implementation is implemented using hardware elements and / or software elements may vary according to any number of factors, such as the desired computational speed, power levels, thermal tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints.

[0174] One or more aspects of at least one implementation may be implemented by typical instructions stored on a machine-readable medium representing various logic in the processor that, when read by a machine, causes the machine to fabricate logic for performing the techniques described herein. Such representations, known as "IP cores," may be stored on a tangible machine-readable medium and delivered to various customers or manufacturing facilities for loading into the fabrication machines that actually manufacture the logic or processor.

[0175] Although certain features set forth herein have been described with respect to various implementations, this description is not intended to be construed in a limiting sense. Consequently, various modifications of the implementations described herein, as well as other implementations that would be apparent to those skilled in the art to which this disclosure pertains, are within the spirit and scope of this disclosure.

[0176] The following examples concern further implementations.

[0177] A computer-implemented method of camera lens distortion correction comprises receiving location data of pixels on at least one distorted image of a scene and distorted by a wide-angle camera lens; receiving pixel locations of an undistorted pixel grid to be used to form an undistorted image depicting the scene; determining radii extending along a distortion center and to each individual pixel location on the distorted image or pixel locations on the undistorted pixel grid; obtaining a ratio between distorted and undistorted radius values ​​by using at least one of the radii to look up the ratio on a lookup table (LUT);determining a correspondence between one of the pixel locations and a corresponding one of the pixel locations by using the ratio, and repeating with different pixel locations or pixel locations to form a plurality of correspondences; and constructing the undistorted image by using the correspondences to place image data on the undistorted pixel locations.

[0178] In another implementation, the method may include the radii extending from the distortion center on the distorted image and extending to individual pixel locations on the distorted image; wherein the radii extend from the distortion center projected onto the undistorted pixel grid and extending to pixel locations on the undistorted pixel grid; and wherein the input values ​​to the LUT are the square of the radii. The method also includes forming a plurality of LUTs, each for a wide-angle lens with a different field of view; storing at least one of the LUTs in a memory accessible to a processor that applies distortion correction to images from a camera using a lens associated with the LUT; and at least one of the following: (1) forming a single LUT for a particular field of view angle, and (2) forming the LUT by using a spline operation.The method may also include obtaining the correspondence separately in both a horizontal and a vertical direction to provide 2D coordinates of the image by using the ratio separately in two equations; applying a scaling value in one of the equations if the distortion in the horizontal and vertical directions is not the same.

[0179] The method may also include using radius ratios to determine the correspondences being called lens distortion correction (LDC); and performing a homography-based, multi-coefficient, multi-degree-of-freedom distortion correction either (a) before performing the LDC to provide modified pixel locations to be used as the undistorted pixel locations to determine radii of the pixel locations during performing the LDC, or (b) after performing the LDC to further modify the correspondence between pixel locations and the pixel locations on the undistorted image.The method may further comprise that the undistorted pixel locations placed on the corresponding pixel locations on the undistorted image are anchor points, and that the method comprises interpolation points between the anchor points on the undistorted image; using an image processing unit other than a central processing unit (CPU) to perform the interpolation of distortion correction operations; dividing the images into blocks; and performing the distortion correction by image processing hardware other than a central processing unit and having a capacity set in association with a single block size.

[0180] In yet another implementation, a camera lens distortion correction system comprises at least one memory; at least one processor communicatively coupled to the memory; and a distortion correction unit operated by the at least one processor unit to operate by: receiving location data of pixels of at least one distorted image of a scene and distorted by a wide-angle camera lens; receiving pixel locations of an undistorted pixel grid; performing a first correction operation using the pixel locations and pixel locations to generate intermediate pixel locations; performing a second correction operation using the intermediate pixel locations to generate modified pixel locations to be used to form an undistorted image of the scene;performing homography-based distortion correction with multiple homography coefficients as one of the first and second operations; and performing non-homography lens distortion correction (LDC).

[0181] In another example, the system comprises that the homography is performed either before or after the LDC; and that performing the homography-based distortion correction comprises performing reverse homography calculations using homography coefficient equations that use undistorted pixel grid locations as an input to the equations to calculate corresponding distorted pixel locations. The system having the distortion correction unit that operates by dividing the image into blocks; obtaining predetermined homography coefficients to be used with a particular camera and with a particular lens on the camera; applying homography coefficients to at least one other block that are different from the homography coefficients of at least one other block; and while performing both the first and second correction operations block by block.The system also includes at least one processing unit that performs the distortion correction being image processing hardware other than a central processing unit and having a capacity set in association with a single block size; and wherein the distortion correction unit operates by calculating a tangential distortion component of at least one of the following: (a) added to resulting coordinate values ​​of the distorted input image coordinates determined after applying LUT-based LDC, and (b) used to adjust homography coefficients; wherein performing the LDC is based at least in part on ratios of radii extending from a distortion center to both individual pixel locations on the distorted image and pixel locations on the undistorted pixel grid.The system also comprises that the ratios are obtained from at least one predetermined lookup table.

[0182] In one approach, at least one computer-readable medium comprises a plurality of instructions that, in response to execution on a computing device, cause the computing device to operate by: receiving location data of pixels of at least one distorted image of a scene distorted by a wide-angle camera lens; receiving pixel locations of an undistorted pixel grid; performing a first correction operation using the pixel locations and pixel locations to generate intermediate pixel locations; performing a second correction operation using the intermediate pixel locations to generate modified pixel locations to be used to form an undistorted image of the scene; performing a homography-based distortion correction having a plurality of homography coefficients as one of the first and second operations;and providing the option to perform either (1) or (2) depending on a size of a field of view of a wide-angle lens forming the distorted image as the other of the first and second operations, where (1) is a non-homography lens distortion correction (LDC), and (2) is a dewarping correction that converts spherical coordinates to Cartesian coordinates. In another approach, the instructions include the LDC using a radius ratio lookup table.

[0183] In another example, at least one machine-readable medium may include a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform the method of any of the above examples.

[0184] In yet another example, a device may include means for performing the methods of any of the above examples.

[0185] The above examples may include specific combinations of features. However, the above examples are not limited in this regard, and the above examples may include performing only a subset of such features, performing a different order of such features, performing a different combination of such features, and / or performing additional features beyond those explicitly listed. Any features described with respect to any example method herein may be implemented with respect to any example device, example systems, and / or example articles, and vice versa.

Claims

[1] A computer-implemented method (1000) for camera lens distortion correction, comprising: Receiving (1002) location data of pixels on at least one distorted image of a scene and distorted by a wide-angle camera lens; Receiving (1004) pixel locations of an undistorted pixel grid for use in forming an undistorted image showing the scene; Determining (1006) radii extending from a distortion center and either to individual pixel locations on the distorted image or to pixel locations on the undistorted pixel grid; Obtaining (1008) a ratio between distorted and undistorted radius values by using at least one of the radii to look up the ratio on a lookup table, LUT; Determining (1010) a correspondence between one of the pixel locations and a corresponding one of the image point locations by using the ratio and repeating with different pixel locations or image point locations to form a plurality of correspondences; and Constructing (1012) the undistorted image by using the plurality of correspondences to place image data on the undistorted pixel layers, the method further comprising dividing the image into blocks, obtaining predetermined homography coefficients to be used with a particular camera and with a particular lens on the camera, and applying homography coefficients to at least one block that are different from the homography coefficients of at least one other block. [2] The method of claim 1, wherein the radii extend from the distortion center of the distorted image and extend to individual pixel locations of the distorted image. [3] A method according to claim 1 or 2, wherein the radii extend from the distortion center projected onto the undistorted pixel grid and extend to pixel locations on the undistorted pixel grid. [4] A method according to any one of claims 1 to 3, wherein the input values to the LUT are the square of the radii. [5] A method according to any one of claims 1 to 4, comprising forming a plurality of LUTs each for a wide-angle lens of a different field of view; and storing at least one of the LUTs on a memory accessible to a processor that performs distortion correction on images from a camera using a lens associated with the LUT. [6] A method according to any one of claims 1 to 5, comprising: Obtaining the correspondence separately in both a horizontal and a vertical direction to provide 2D coordinates of the image by using the ratio separately in two equations; and Apply a scaling value in one of the equations when the distortion in the horizontal and vertical directions is not the same. [7] The method of any one of claims 1 to 6, wherein using radius ratios to determine the correspondences is called a lens distortion correction, LDC; and the method comprises performing a homography-based distortion correction with multiple coefficients of multiple degrees of freedom prior to performing the LDC to provide modified pixel locations that are used as the undistorted pixel locations to determine radii of the pixel locations during performing the LDC. [8] The method of any one of claims 1 to 7, wherein using radius ratios to determine the correspondences is called lens distortion correction, LDC; and the method comprising performing a homography-based distortion correction with multiple coefficients of multiple degrees of freedom after performing the LDC to further modify the correspondence between pixel locations and the pixel locations on the undistorted image. [9] A method according to any one of claims 1 to 8, wherein the undistorted pixel locations placed on the corresponding pixel locations on the undistorted image are anchor points, and wherein the method comprises interpolation points between the anchor points on the undistorted image. [10] A method according to any one of claims 1 to 9, comprising using an image processing unit other than the central processing unit, CPU, to perform the interpolation of distortion correction operations. [11] A method according to any one of claims 1 to 10, comprising dividing the images into blocks; and performing the distortion correction by image processing hardware other than a central processing unit, and having a capacity set in association with a size of a single block. [12] Camera lens distortion correction system comprising: at least one memory; at least one processor communicatively coupled to the memory; and a distortion correction unit operated by the at least one processor unit to operate by: Receiving location data of pixels of at least one distorted image of a scene and distorted by a wide-angle camera lens; Receiving pixel layers of an undistorted pixel grid; performing a first correction operation using the image point positions and pixel positions to generate intermediate image point positions; performing a second correction operation using the intermediate pixel locations to generate modified pixel locations to be used to form an undistorted image of the scene; Performing a homography-based distortion correction with multiple homography coefficients as one of the first and second operations; and performing a non-homography lens distortion correction, LDC; the system further comprising dividing the image into blocks; obtaining predetermined homography coefficients to be used with a particular camera and with a particular lens on the camera; and applying homography coefficients to at least one block that are different from the homography coefficients of at least one other block. [13] The system of claim 12, wherein the homography is performed before the LDC. [14] A system according to claim 11 or 12, wherein the homography is performed after the LDC. [15] The system of any one of claims 12 to 14, wherein performing the homography-based distortion correction comprises performing inverse homography calculations using homography coefficient equations that use undistorted pixel grid locations as an input to the equations to calculate corresponding distorted pixel locations. [16] A system according to any one of claims 12 to 15, comprising: Dividing the images into blocks; and block-wise execution of both the first and second correction operations. [17] The system of any one of claims 12 to 16, wherein the at least one processing unit that performs the distortion correction is image processing hardware other than a central processing unit and has a capacity set in association with a size of a single block. [18] A system according to any one of claims 12 to 17, wherein the distortion correction unit operates by calculating a tangential distortion component of at least one of the following: added to resulting coordinate values of the distorted input image coordinates determined after applying LUT-based LDC and used to adjust homography coefficients. [19] The system of any of claims 12 to 18, wherein performing the LDC is based at least in part on ratios of radii extending from a distortion center to both individual pixel locations on the distorted image and pixel locations on the undistorted pixel grid. [20] The system of claim 19, wherein the ratios are obtained from at least one predetermined lookup table. [21] A machine-readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to operate by: Receiving (1052) location data of pixels of at least one distorted image of a scene and distorted by a wide-angle camera lens; Receiving (1054) pixel positions on an undistorted pixel grid; performing (1056) a first correction operation using the pixel positions and pixel positions to generate intermediate pixel positions; performing (1058) a second correction operation using the intermediate pixel locations to generate modified pixel locations to be used to form an undistorted image of the scene; performing (1060) a homography-based distortion correction with a plurality of homography coefficients as one of the first and second operations; Providing the option of performing either (1) or (2) depending on a size of a field of view of a wide-angle lens forming the distorted image as the other of the first and second operations, wherein (1) is a non-homography lens distortion correction, LDC, and (2) is a dewarping correction that converts spherical coordinates into Cartesian coordinates; Dividing the image into blocks; Obtaining predetermined homography coefficients to be used with a particular camera and with a particular lens on the camera; and Applying homography coefficients to at least one block that are different from the homography coefficients of at least one other block. [22] The machine-readable medium of claim 21, wherein the LDC uses a radius ratio lookup table. [23] A machine-readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform the method of any one of claims 1 to 11. [24] Apparatus comprising means for carrying out the method according to any one of claims 1 to 11.

Images