Image de-warping system

Abstract

An image processing method and apparatus. A de-warping circuit is configured to retrieve, in an ordered sequence, a plurality of de-warping matrices for remapping pixels of a first image from a distorted image space to a corrected image space, where the first image depicts a scene in the distorted image space. The de-warping circuit reads, into a memory buffer, blocks of pixels of the first image based at least in part on the order of the de-warping matrices, and generates a plurality of image tiles based on the blocks of pixels of the first image. Each of the image tiles is interpolated from one or more of the retrieved blocks using a respective one of the de-warping matrices. The de-warping circuit writes the plurality of image tiles to an external memory to produce a second image that depicts the scene in the corrected image space.

Description

Technical Field

[0001] This embodiment generally involves image processing. Background Technology

[0002] Image processing enables the rendering of captured images on a display, allowing for the reproduction of the original scene as accurately as possible, taking into account the capabilities or limitations of the image capture device (e.g., a camera). For example, a wide-angle lens can be used to capture a larger field of view (FOV) than a standard linear lens. However, wide-angle lenses also introduce geometric nonlinear distortions into the captured image. Examples of geometric distortions include radial or barrel distortion, tangential distortion, etc. Therefore, image processing can correct geometric distortions in the captured image, making the scene appear more linear. The process of reversing the geometric distortions introduced by the lens optics and / or the viewing angle of the image capture device is often referred to as dewarping. Summary of the Invention

[0003] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed embodiments. The present invention is not intended to identify key or essential features of the subject matter of the claims, nor is it intended to limit the scope of the claimed subject matter.

[0004] A method and apparatus for image processing are disclosed. An innovative aspect of the subject matter of this disclosure can be implemented in an image processing method. In some embodiments, the method may include the steps of: retrieving, in an ordered sequence, a plurality of dedistortion matrices for remapping pixels of a first image from a distorted image space to a corrected image space, wherein the first image depicts a scene in the distorted image space; retrieving pixel blocks of the first image at least in part based on the order of the dedistortion matrices; generating a plurality of image tiles based on the pixel blocks of the first image; wherein each of the image tiles is interpolated from one or more of the retrieved blocks using a corresponding one of the dedistortion matrices; and combining the plurality of image tiles to generate a second image depicting a scene in the corrected image space.

[0005] Another innovative aspect of the subject matter of this disclosure can be implemented in an image processing system. In some embodiments, the system may include a processor, a memory, and de-distortion circuitry. The memory stores instructions that, when executed by the processor, cause the system to generate a plurality of de-distortion matrices in an ordered sequence for remapping pixels from a distorted image space to a corrected image space.

[0006] The de-distortion circuit is configured to retrieve pixel blocks of a first image, at least in part, based on the order of the de-distortion matrix, wherein the first image depicts a scene in a distorted image space; generate a plurality of image patches based on the pixel blocks of the first image, wherein each image patch is interpolated from one or more of the retrieved pixel blocks using a corresponding one in the de-distortion matrix; and combine the plurality of image patches to produce a second image depicting a scene in a corrected image space. Attached Figure Description

[0007] This embodiment is shown by way of example and is not intended to be limited to the figures in the accompanying drawings.

[0008] Figure 1 A block diagram of an image capture and display system according to some embodiments is shown.

[0009] Figure 2 A block diagram of a lens geometry distortion correction (LGDC) circuit according to some embodiments is shown.

[0010] Figure 3 An example of correcting image space is shown according to some embodiments.

[0011] Figure 4 An example pixel mapping between the corrected image space and the distorted image space is shown.

[0012] Figure 5 A block diagram of an LGDC system according to some embodiments is shown.

[0013] Figure 6 A block diagram of an image space remapping controller according to some embodiments is shown.

[0014] Figure 7 A block diagram of a block de-twisting circuit according to some embodiments is shown.

[0015] Figure 8 Another block diagram of an image space remapping controller according to some embodiments is shown.

[0016] Figure 9 This is an illustrative flowchart depicting example image processing operations according to some embodiments. Detailed Implementation

[0017] In the following description, numerous specific details, such as examples of specific components, circuits, and processes, are set forth to provide a thorough understanding of this disclosure. As used herein, the term "coupling" means a direct connection to or a connection via one or more intermediate components or circuits. Additionally, specific terminology is set forth in the following description and for purposes of explanation to provide a thorough understanding of aspects of this disclosure. However, it will be apparent to those skilled in the art that practical example embodiments may not require these specific details. In other instances, well-known circuits and devices are illustrated in block diagram form to avoid obscuring this disclosure. Some portions of the following detailed description are presented according to procedures, logic blocks, processes, and other symbolic representations of operations on data bits within computer memory. Interconnections between circuit elements or software blocks may be shown as buses or single signal lines. Each bus may alternatively be a single signal line, and each single signal line may alternatively be a bus, and a single line or bus may represent any one or more of the numerous physical or logical mechanisms used for communication between components.

[0018] Unless otherwise expressly stated (as is evident from the discussion below), it should be understood that throughout this application, discussions using terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “standardize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” and “derive” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memories of the computer system into other data represented as physical quantities in the memory or registers of the computer system or other such information storage, transmission, or display devices.

[0019] Unless explicitly described as being implemented in a particular manner, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Any feature described as a module or component may also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory computer-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory computer-readable storage medium may form part of a computer program product, which may include encapsulation material.

[0020] Non-transitory processor-readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Additionally or alternatively, this technology may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures, and can be accessed, read, and / or executed by a computer or other processor.

[0021] The various illustrative logic blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein can be executed by one or more processors. As used herein, the term "processor" can refer to any general-purpose processor, conventional processor, controller, microcontroller, special-purpose processor, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0022] Figure 1 A block diagram of an image capture and display system 100 according to some embodiments is shown. System 100 includes an image capture device 110, an image processor 120, and an image display device 130. The image capture device 110 captures a pattern of light 101 from a scene and converts the captured light 101 into digital image capture data 102. The image capture data 102 may correspond to a digital image depicting the scene. The image display device 130 displays the digital image by reproducing the light pattern on a corresponding display surface. In some aspects, the image capture device 110 may be a camera, and the image display device 130 may be an electronic display (such as a television, computer monitor, smartphone, etc.).

[0023] Image processor 120 performs image processing on image capture data 102 to generate image rendering data 103, which can be used to more accurately reproduce the original scene (captured by image capture device 110) on image display device 130. In some embodiments, image processor 120 may be incorporated into or otherwise included in image capture device 110. Image processor 120 includes image signal processing (ISP) module 122 and lens geometric distortion correction (LGDC) module 124. ISP module 122 can correct various pixel distortions in image capture data 102 to improve the quality of the digital image. Example pixel distortions include, but are not limited to, vignetting, aberrations, and noise.

[0024] The LGDC module 124 can transform or transfer the image capture data 102 to a different image space that is more suitable or optimized for display on the image display device 130. For example, a wide-angle lens on the image capture device may introduce geometric distortions (such as radial or barrel distortion, tangential distortion, trapezoidal distortion, etc.) into the scene depicted in the image capture data 102. Therefore, the LGDC module 124 can be configured to correct the geometric distortions introduced by the image capture device 110, so that the captured image or scene appears more linear on the image display device 130. In some embodiments, image rendering data 103 can be provided to other devices besides or instead of the image display device 130. For example, computer vision processing can benefit from linear images. Examples of computer vision processing may include, but are not limited to, face detection, object detection, and object tracking.

[0025] This disclosure recognizes that certain applications may require near real-time processing of image capture data 102. Examples of real-time applications include, but are not limited to, streaming video applications (such as those from content delivery networks, video surveillance systems, video conferencing systems, etc.) and computer vision applications (such as facial recognition systems, automotive driver assistance systems, augmented reality (AR) systems, etc.). However, LGDC operation requires complex computation and data manipulation. A complete software dewarping solution could consume a significant portion of the bandwidth or processing power of the central processing unit (CPU). On the other hand, a complete hardware dewarping architecture could occupy a significant footprint and may offer limited flexibility to reconfigure or customize the dewarping algorithm for different lens geometries.

[0026] In some embodiments, the image processor 120 may implement the LGDC module 124 as a combined hardware and software solution. More specifically, aspects of this disclosure may leverage the flexibility and scalability of software (e.g., executed by a CPU or dedicated processor) to generate instructions for remapping one or more pixels of image capture data 102 to one or more pixels of image rendering data 103 based on lens parameters of the image capture device 110. Aspects of this disclosure may further leverage the speed and bandwidth of hardware logic to perform the computations and data manipulations articulated in the instructions generated by the CPU.

[0027] In some embodiments, the image space of the image rendering data 103 (e.g., the "corrected image space") can be segmented or subdivided into multiple tiles, allowing for more fine-grained de-distortion operations. For example, the size of each tile can be configured or optimized based at least in part on the storage bandwidth of the LGDC module 124. Performing pixel remapping operations at the tile granularity significantly reduces the hardware overhead required to de-distort frames or images of the image capture data 102. This can further reduce image processing latency or delay, enabling near real-time generation of the image rendering data 103.

[0028] Figure 2 A block diagram of a lens geometric distortion correction (LGDC) circuit 200 according to some embodiments is shown. The LGDC circuit 200 may be... Figure 1 One embodiment of the LGDC module 124. Therefore, the LGDC circuit 200 can be configured to correct geometric distortion in the distorted image 202. For example, geometric distortion can be introduced by lenses and / or other characteristics of the image capture device used to acquire or otherwise generate the distorted image 202. In some embodiments, the de-distortion circuit 210 can be configured to remap one or more pixels of the distorted image 202 to one or more pixels of the corrected image 204, such that the corrected image 204 appears substantially straight.

[0029] exist Figure 2 In this embodiment, the distorted image 202 may be captured by a camera (or other image capture device) with a wide-angle lens. Therefore, the scene depicted in the distorted image 202 appears radially distorted. For example, objects with straight lines (such as buildings) appear curved in the distorted image 202. In contrast, the scene depicted in the corrected image 204 appears straight. For example, objects with straight lines appear straight in the corrected image 204. As used herein, the term "distorted image space or DIS" refers to the image space of the distorted image 202 (e.g., before dedistortion), and the term "corrected image space or CIS" refers to the image space of the corrected image 204 (e.g., after dedistortion).

[0030] The LGDC circuit 200 includes a de-distortion circuit 210 and a memory 220. The de-distortion circuit 210 can receive image data corresponding to the distorted image 202 and can output corrected image data corresponding to the corrected image 204. In some aspects, the corrected image 204 may have the same dimensions as the distorted image 202 (e.g., pixel height and pixel width). However, less of the scene can be depicted in the corrected image 204 compared to the distorted image 202. For example, a building in the upper left corner of the distorted image 202 is not shown in the corrected image 204. Therefore, the de-distortion circuit 210 can interpolate each pixel of the corrected image 204 from one or more pixels of the distorted image 202.

[0031] In some embodiments, the dedistortion circuit 210 may determine the remapping of pixels from the distorted image space to the corrected image space based at least in part on one or more camera lens parameters 206 of the image capture device. Example lens parameters 206 may include, but are not limited to, lens geometry and focal length. Because the dedistortion operation reverses the geometric distortion introduced by the camera lens, the pixel remapping may be the inverse of the pixel mapping created by the camera lens. In some aspects, the dedistortion circuit 210 may determine the pixel remapping calculation based on a polynomial lens model. In some other aspects, the dedistortion circuit 210 may determine the pixel remapping calculation based on a non-polynomial lens model.

[0032] Memory 220 can be configured to store or buffer any data required to complete the dedistortion operation. The data stored in memory 220 may include, but is not limited to, camera lens parameters 206, pixel remapping calculations (e.g., matrices), the index order for extracting distorted image patches, image data of the distorted image 202, and image data of the corrected image 204. Because the amount of storage in memory 220 is limited, storing the complete pixel dataset of the entire frame of the distorted image 202 and / or the corrected image 204 may be inefficient (if not impossible). Therefore, in some embodiments, LGDC circuitry 200 may be configured to generate small portions (e.g., patches) of the corrected image 204 at a time. Since each portion of the corrected image 204 is interpolated only from a relatively small subset (e.g., a patch) of pixels from the distorted image 202, aspects of this disclosure can significantly reduce the storage bandwidth requirements of memory 220.

[0033] Figure 3 An example of correcting image space 300 according to some embodiments is shown. Correcting image space 300 can be... Figure 2 An embodiment of the image space of the corrected image 204. For example... Figure 3As shown, the corrected image space 300 is divided or subdivided into multiple patches T1(1)-Tn(m) of equal or uniform size. Each patch may include multiple pixels of the corrected image space 300. In some embodiments, the size of each patch (e.g., pixel height and pixel width) may be at least partially based on LGDC circuitry (such as...) Figure 2 The storage bandwidth of the memory 220. In some other embodiments, the size of each tile may be at least partially based on the corrected image (such as... Figure 2 The pixel depth and / or color format of the corrected image (204).

[0034] Each row of tiles can form a corresponding stride. For example, stride S1 may include tiles T1(1)-T1(m), stride S2 may include tiles T2(1)-T2(m), stride S3 may include tiles T3(1)-T3(m), and so on. Therefore, each of strides S1-Sn can span the width of the corrected image space 300 (e.g., the combined pixel width corresponding to tiles T1(1)-T1(m)) and can have a height equal to the pixel height of one of the tiles. In some embodiments, dewarping can be performed on each tile in raster order, one stride at a time. For example, the dewarping circuit can process each tile of stride S1 (e.g., T1(1), T1(2), ..., T1(m)) sequentially from left to right, and then continue processing each tile of stride S2 (e.g., T2(1), T2(2), ..., T2(m)) from left to right.

[0035] The size and dimensions of each step can depend at least in part on the order in which the raster updates are displayed and / or the granularity of each update. Figure 3 In one embodiment, strides S1-Sn are shown corresponding to horizontal rows of the corrected image space 300, for example, to support near real-time dewarping for horizontal display updates. In other embodiments, strides may correspond to vertical columns of the corrected image space 300, for example, to support near real-time dewarping for vertical display updates. For example, the dewarping circuitry may process entire columns of tiles sequentially from top to bottom (e.g., T1(1), T2(1), ..., Tn(1)), and then continue processing the next column of tiles from top to bottom (e.g., T1(2), T2(2), ..., Tn(2)). Furthermore, in some embodiments, multiple strides of the corrected image space 300 may be processed simultaneously (e.g., using multithreading).

[0036] As will be appreciated by all aspects of this disclosure, the pixel mapping between the corrected image space 300 and the distorted image space can be non-linear. For example, see reference... Figure 2Geometric distortion may be more pronounced at the edges or periphery of the distorted image 202 compared to the center of the image. Therefore, pixels at the center of the corrected image 204 can be mapped to pixels at the center of the distorted image 202. However, pixels at the corners and edges of the corrected image 204 may not be mapped to corresponding pixels at the corners and edges of the distorted image 202. Instead, pixels at the corners and edges of the corrected image 204 can be mapped to pixels closer to the center of the distorted image 202.

[0037] Figure 4 An example pixel mapping 400 between the corrected image space 410 and the distorted image space 420 is shown. The corrected image space 410 can be... Figure 3 An embodiment of the corrected image space 300. In Figure 4 In this embodiment, distortion is relatively inconspicuous at the center of the distorted image space 420. For example, patch 412 at the center of the corrected image space 410 can be mapped to pixel patch 422 at the center of the distorted image space 420. However, distortion becomes more pronounced towards the edges of the distorted image space 420. For example, patches 414, 416, and 418 at the top left corner of the corrected image space 410 can each be mapped to the same pixel patch 424 in the distorted image space 420.

[0038] Therefore, in some embodiments, a corresponding dedistortion matrix (or set of matrices) can be generated for each tile of the corrected image space 410. Each dedistortion matrix can define a pixel mapping between a specific tile of the corrected image space 410 and a corresponding pixel block of the distorted image space 420. In some embodiments, the size (e.g., pixel height and pixel width) of each pixel block in the distorted image space 420 can be based at least in part on the size of each tile of the corrected image space 410 and / or the storage bandwidth of the LGDC circuit. In some aspects, the block size of the corrected image space 410 and the tile size of the distorted image space 420 can be configured at least in part based on the tile granularity of the memory storage format. More specifically, the tile granularity can limit the size of the blocks and / or tiles depending on the image format and pixel depth of the image stored in memory.

[0039] In some embodiments, the relationship between the block size of the corrected image space 410 and the block size of the distorted image space 420 may also vary relative to one or more lens distortion parameters. For example, when the distorted image space 420 represents a field of view that is substantially wider than that of the corrected image space 410, the block size of the distorted image space 420 may be significantly larger than the block size of the corrected image space 410. However, for less significant distortions, the block size of the distorted image space 420 may be closer to the block size of the corrected image space 410. In some aspects, the blocks of the distorted image space 420 may be larger than the blocks of the corrected image space 410 (e.g., at least twice their size). As described in more detail below, this can reduce the number and / or frequency of block read operations (e.g., from memory) required to process each block of the corrected image space 410.

[0040] Figure 5 A block diagram of an LGDC system 500 according to some embodiments is shown. The LGDC system 500 may be... Figure 2 This is one embodiment of the LGDC circuit 200. Therefore, the LGDC system 500 can be configured to correct geometric distortion in image capture data 501. The LGDC system 500 includes an image spatial remapping controller 510, a memory 520, and a tile de-distortion circuit 530.

[0041] Image space remapping controller 510 can be configured to generate remapping data 502 for remapping pixels of image capture data 501 from distorted image space to corrected image space. In some embodiments, remapping data 502 may include multiple de-distortion matrices and multiple lookup tables (LUTs) identifying pixel blocks of image capture data 501 to be used by the de-distortion matrices. (See also: Regarding...) Figure 3 and Figure 4 As described, each dedistortion matrix can define a pixel mapping between a corresponding patch in the corrected image space and one or more pixel blocks in the distorted image space. Therefore, a dedistortion matrix can define a series of dedistortion operations (e.g., computations) to be performed by the dedistortion circuit 530.

[0042] The image spatial remapping controller 510 can generate a dedistortion matrix based at least in part on lens configuration data 503 associated with the image capture device used to capture image capture data 501. For example, the lens configuration data 503 may indicate one or more lens parameters (such as lens geometry and focal length) of the image capture device. In some aspects, the lens configuration data 503 may be provided to the remapping controller 510 during the device manufacturing stage. In some other aspects, the lens configuration data 503 and / or new lens configuration data may be provided to the remapping controller 510 at any time after the LGDC system 500 is manufactured. Therefore, the LGDC system 500 can be configured (and reconfigured) to dedistort images captured by various image capture devices.

[0043] The LUT can specify an order in which the de-distorting circuit 530 retrieves and operates on the pixel blocks of the image capture data 501. The order of the blocks identified by the LUT can be based at least in part on the order of the de-distorting matrix to be implemented by the de-distorting circuit 530. More specifically, the LUT can ensure that the de-distorting circuit 530 retrieves the necessary pixel blocks in the correct order to interpolate each patch of the corrected image space. In some embodiments, the de-distorting matrix can be defined by the de-distorting circuit 530 in raster order (e.g., as mentioned above regarding...). Figure 3 (As described) the processing. Therefore, the order of blocks specified by the LUT can also depend on the raster order of the tiles in the corrected image space.

[0044] Memory 520 can be configured to store or buffer any data required to complete the dedistortion operation. In some embodiments, memory 520 can be dynamic random access memory (DRAM). Memory 520 may include remapping data storage 522, distortion image space (DIS) image data storage 524, and correction image space (CIS) image data storage 526. Remapping data storage 522 can store or buffer remapping data 502 from a remapping controller. DIS image data storage 524 can store or buffer image capture data 501. In some aspects, DIS image data storage 524 can be configured to store full-frame pixel data depicting the scene in the distortion image space. CIS image data storage 526 can store or buffer data to be output to another device or electronic system (such as...). Figure 1 Image rendering data 507 of the image display device 130. In some aspects, the CIS image data storage 526 can be configured to store full-frame pixel data depicting a scene in the corrected image space.

[0045] The de-distortion circuit 530 can be configured to generate image rendering data 507 based on image capture data 501. For example, the de-distortion circuit 530 can interpolate each pixel of the image rendering data 507 from one or more pixels of the image capture data 501 (e.g., using bicubic interpolation). In some embodiments, the de-distortion circuit 530 can operate at the per-tile granularity of the corrected image space. More specifically, the de-distortion circuit 530 can process each tile of the corrected image space sequentially based on the order (e.g., raster order) of the de-distortion matrices stored in the remapping data storage 522.

[0046] The de-distorting circuit 530 can retrieve remapping data 502 from the remapping data store 522 to determine a de-distorting matrix associated with each tile in the corrected image space. The de-distorting circuit 530 can also determine the pixel blocks to be operated on (e.g., image capture data 501) based on the LUTs included in the remapping data 502. For example, the de-distorting circuit 530 can retrieve one or more DIS image blocks 504 from the DIS image data store 524 in the order specified by the LUTs. The de-distorting circuit 530 can perform bicubic interpolation on the DIS image blocks 504 defined by the de-distorting matrix to produce one or more CIS image tiles 506. The de-distorting circuit 530 can then store the CIS image tiles 506 in the CIS image data store 526 as image rendering data 507. Therefore, the CIS image tiles 506 can be combined (e.g., in raster order) to produce a full-frame image depicting the scene of the image capture data 501 in the corrected image space.

[0047] In some embodiments, the remapping controller 510 may be implemented in software. For example, the remapping controller 510 may include a set of instructions executed by a general-purpose processor or CPU. This provides greater scalability and flexibility in the design and / or configuration of the dedistortion matrix. For example, the algorithm for reversing geometric distortion of a camera lens may be user-configurable (and reconfigurable). Thus, aspects of this disclosure can support dedistortion matrices based on polynomial or non-polynomial lens models (e.g., for correcting radial or tangential distortion). In some other aspects, the dedistortion matrix may be configured or updated to support additional geometric distortion corrections, including but not limited to tilt, scaling, rotation, and focus changes.

[0048] In some embodiments, the de-distortion circuit 530 may be implemented in hardware. For example, the de-distortion circuit 530 may include a set of transistors or logic gates hardwired to perform calculations or mathematical operations defined by the de-distortion matrix, such as bicubic interpolation or other pixel remapping functions. This provides greater speed and bandwidth when interpolating image rendering data 507. By processing image rendering data 507 at a tile-size granularity, aspects of this disclosure can further reduce the latency of the de-distortion process while also optimizing the use of available bandwidth and resources. Therefore, frames of image rendering data 507 can be generated near real-time (e.g., enabling the LGDC system 500 to output image rendering data 507 at substantially the same rate as the system 500 receives image capture data 501).

[0049] Figure 6 A block diagram of an image space remapping controller 600 according to some embodiments is shown. The remapping controller 600 may be... Figure 5 One embodiment of the image space remapping controller 510. Therefore, the remapping controller 600 can be configured to generate the image space remapping controller 600 to be used for remapping. Figure 5 The image capture data 501 contains pixels that have been remapped from the distorted image space to the corrected image space. In some embodiments, the remapping controller 600 may generate a stride remapping dataset 604 for each stride 602 of the corrected image space (CIS). The remapping controller 600 includes a dedistortion matrix generator 610 and an image patch LUT generator 620.

[0050] The dedistortion matrix generator 610 is configured to generate a plurality of dedistortion matrices M(1)-M(m) based at least in part on lens configuration data 603 associated with the image capture device. Each of the dedistortion matrices M1(1)-M(m) may be associated with a corresponding CIS matrix patch T(1)-T(m) of the current CIS matrix stride 602. Matrix elements in each CIS matrix patch T() may refer to the row and column indices of CIS pixels. In some embodiments, the dedistortion matrices M(1)-M(m) may define a pixel mapping between CIS pixels and pixels in the Distortion Image Space (DIS). Therefore, matrix elements in each dedistortion matrix M() may refer to the row and column indices of DIS pixels.

[0051] In some embodiments, the matrix generator 610 can process each of the CIS matrix tiles T(1)-T(m) in raster order (e.g., as mentioned above regarding...). Figure 3(as described above). In some aspects, each dedistorting matrix element may include a high-precision pixel reference to a pixel in the referenced DIS block. When processing each pixel of a given CIS matrix patch, the associated dedistorting matrix element also specifies one or more fractional coordinates for intra-pixel DIS pixel references from the DIS block, from which pixels are interpolated.

[0052] Image block LUT generator 620 is configured to generate a corresponding block lookup table (LUT) for each CIS matrix strut 602. Each LUT can identify one or more DIS blocks associated with the CIS matrix strut 602. Because memory buffer space may be limited, the LUT can specify where DIS blocks will be retrieved for de-twisting circuitry (such as...). Figure 5 The order in which the block de-distortion circuit 530 processes the blocks. More specifically, the order of the DIS blocks can be aligned with the order of the de-distortion matrices M(1)-M(m), so that the necessary DIS blocks can be prefetched for processing each of the CIS matrix blocks T(1)-T(m).

[0053] In some embodiments, the LUT generator 620 may determine the DIS blocks to be included in the LUT based at least in part on pixel references included in each of the dedistortion matrices M(1)-M(m). For example, the LUT generator 620 may determine regions of distorted image space bounded by the largest and smallest pixel references in a particular dedistortion matrix (or set of matrices). The LUT generator 620 may also identify one or more DIS blocks bounded by (or coinciding with) the identified regions and associate the DIS blocks with the dedistortion matrices. For example, each DIS block in the LUT may be identified by the DIS block column and row offset addresses along with a count or other indicator of the CIS matrix it serves.

[0054] The LUT can be appended to the de-distortion matrix M(1)-M(m) and stored in system memory (such as stride remapping dataset 604). Figure 5 The LUT is stored in memory 520. In some embodiments, the LUT may be compressed (e.g., using run-length encoding) to reduce the storage overhead of the dataset 604. In some other embodiments, the remapping matrices M(1)-M(m) may also be compressed to further reduce storage overhead and / or alleviate bandwidth congestion. The LUT generator 620 may process each stride 602 of the corrected image space sequentially (e.g., in raster order) for each frame of the image capture data. In some aspects, the LUT generator 620 may output the remapping dataset 604 for each stride as a continuous data stream.

[0055] Figure 7A block diagram of a block de-twisting circuit 700 according to some embodiments is shown. The de-twisting circuit 700 may be... Figure 5 This is one embodiment of the tile de-distortion circuit 530. Therefore, the de-distortion circuit 700 can be configured to generate corrected image data based on the distorted image data. In some embodiments, the de-distortion circuit 700 can process each tile in the corrected image space individually (e.g., in raster scan order) by incorporating block pixel data 708 from corresponding block pixel data 706 in the distorted image space.

[0056] The de-distortion circuit 700 includes a memory interface 710, a stride processing controller 720, a distortion image space (DIS) input buffer 730, a corrected image space (CIS) output buffer 740, an interpolator 750, and a timing controller 760. The memory interface 710 provides an interface through which the de-distortion circuit 700 can communicate with system memory (such as…). Figure 5 The memory interface 710 communicates with the memory 520. The memory interface 710 may include a stride remapping (SR) data read client 712, a DIS image block read client 714, and a CIS image block write client 716. The stride remapping data read client 712 may read a stride remapping (SR) dataset 702 from the memory in response to a stride read request 701. The DIS image block read client 714 may read block pixel data 706 from the memory in response to a block read request 705. The CIS image block write client 716 may write block pixel data 708 to the memory in response to a block write request 707. In some aspects, the CIS image block write client 716 may write the block pixel data 708 to the memory in raster scan order.

[0057] The stride processing controller 720 can process the stride in the corrected image space. (See also: Regarding...) Figure 3 As described, the corrected image space may include multiple strides having a predetermined pixel height and spanning the width of the corrected image space. The stride processing controller 720 may output a corresponding stride read request 701 for processing each stride of the corrected image space. The stride remapping data read client 712, in response to the stride read request 701, returns the SR dataset 702 to the stride processing controller 720. (See also: [link to relevant documentation]). Figure 6As described, the SR dataset 702 may include block LUTs and multiple de-distortion matrices M(1)-M(m). Each de-distortion matrix may define a pixel mapping between a corresponding tile in the corrected image space and one or more DIS blocks. The LUT may identify the DIS block serving each de-distortion matrix. The stride processing controller 720 may store the block LUTs and de-distortion matrices in LUT buffer 722 and matrix buffer 724, respectively. In some embodiments, matrix buffer 724 may be configured to store only a single de-distortion matrix at a time.

[0058] The DIS input buffer 730 can read block information 703 from the LUT buffer 722 to determine which DIS blocks to retrieve from memory. For example, the block information 703 can be parsed from the LUT and provided to the DIS input buffer 730 in the order in which the de-distortion matrices are to be processed. The block information 703 may include indices of one or more DIS blocks to be read from memory and count values ​​indicating the number of de-distortion matrices referencing each DIS block. The DIS input buffer 730 may output a corresponding block read request 705 for each DIS block (or set of DIS blocks) with the index specified in the current block information 703. The DIS image block read client 714 returns the block pixel data 706 of the requested DIS blocks(s) to the DIS input buffer 730. The DIS input buffer 730 may store or buffer the block pixel data 706 until subsequent block information 703 is read from the LUT buffer 722.

[0059] Interpolator 750 can process each tile in the corrected image space based on block pixel data 706 and the corresponding de-distortion matrix 704. For example, interpolator 750 can read each de-distortion matrix 704 from matrix buffer 724 in raster order. For each de-distortion matrix 704, interpolator 750 can use the buffered block pixel data 706 to generate a corresponding tile in the corrected image space according to the pixel mapping specified by the de-distortion matrix 704. In some embodiments, interpolator 750 can generate each tile based on bicubic interpolation of a reference point indicated by the de-distortion matrix in the block pixel data 706. For example, interpolator 750 can interpolate each tile using four luma (Y) pixels and four chroma (UV) pixels of the block pixel data 706 and a fractional offset indicated by the de-distortion matrix 704.

[0060] The CIS output buffer 740 can buffer the output of the interpolator 750 (e.g., tile pixel data 708) until a complete tile is stored in the CIS output buffer 740. When the tile pixel data 708 in the CIS output buffer 740 represents a complete tile in the corrected image space, the CIS output buffer 740 can write the tile back to system memory. For example, the CIS output buffer 740 can output a tile write request 707 along with the tile pixel data 708 to the CIS image tile write client 716. In some embodiments, each complete tile can be written to system memory in raster order before the interpolator 750 begins processing the next tile (or the next de-distortion matrix 704) in the corrected image space. Therefore, when the de-distortion circuit 700 completes de-distortion, the image rendering device (or image analysis device) can render the tile pixel data 708 in near real-time.

[0061] The timing controller 760 can generate a timing signal 709 to control the timing of the stride processing controller 720, the DIS input buffer 730, and the CIS output buffer 740. More specifically, the timing signal 709 can be used to synchronize the operation of the stride processing controller 720, the DIS input buffer 730, and the CIS output buffer 740. For example, the timing signal 709 can ensure that the block LUT is loaded into the LUT buffer 722 first, and that the DIS input buffer 730 retrieves the block pixel data 706 identified by the LUT before the stride processing controller 720 retrieves the de-distortion matrix 704 associated with the block pixel data 706. Because multiple de-distortion matrices 704 can depend on the same block pixel data 706, the DIS input buffer 730 does not need to retrieve new block pixel data 706 for each consecutive de-distortion matrix 704. In some aspects, when the number of de-twisted matrices processed by interpolator 750 reaches a count value associated with one or more current DIS blocks stored in DIS input buffer 730, DIS input buffer 730 may output a new block read request 705.

[0062] Figure 8 Another block diagram of an image space remapping controller 800 according to some embodiments is shown. The remapping controller 800 can be used in LGDC systems (such as...) Figure 5 This is implemented in the LGDC system 500. Therefore, the remapping controller 800 can be... Figure 6 One embodiment of an image space remapping controller 600. The remapping controller 800 includes a processor 810 and a memory 820.

[0063] The memory 820 may include an image pixel data storage 821 configured to store image capture data and / or image rendering data. For example, the image capture data may correspond to an image of a scene in a distorted image space, and the image rendering data may correspond to an image of a scene in a corrected image space. The memory 820 may also include a non-transitory computer-readable medium (e.g., one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard disk drive, etc.) capable of storing at least the following software (SW) modules:

[0064] • The dedistortion matrix SW module 822 is used to generate multiple dedistortion matrices representing pixel mappings between patches in the corrected image space (CIS) and patches in the distorted image space (DIS). The dedistortion matrix SW module 822 also includes:

[0065] o Tile configuration submodule 823, for configuring the size of each CIS tile based at least in part on the storage bandwidth utilization of the LGDC system;

[0066] o Polynomial modeling submodule 824 for determining pixel mapping based on polynomial lens model; and

[0067] o Non-polynomial modeling submodule 825 for determining pixel mapping based on a non-polynomial lens model; and

[0068] • Block LUT SW module 826, for generating multiple block LUTs that identify one or more DIS blocks associated with each de-distortion matrix, the block LUT SW module 826 further comprising:

[0069] o Block configuration submodule 827, for configuring the size of each DIS block based at least in part on the storage bandwidth utilization of the LGDC system; and

[0070] The block order submodule 828 is used to determine the extraction order of each DIS block in the DIS block based at least in part on the order of the de-twisting matrix.

[0071] Each software module includes instructions that, when executed by processor 810, cause remapping controller 800 to perform the corresponding function.

[0072] Processor 810 may be any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in remapping controller 800. For example, processor 810 may execute instructions 822-828 to generate a stride remapping dataset (such as...) for each stride in the corrected image space. Figure 6(Dataset 604). In some embodiments, processor 810 may be configured to operate on multiple steps simultaneously or in parallel (e.g., using multithreading) to improve latency. More specifically, software process latency can be reduced by increasing the number of concurrent threads, which are operated on by processor 810.

[0073] Processor 810 can execute a de-distortion matrix SW module 822 to generate multiple de-distortion matrices representing the pixel mapping between CIS tiles and DIS tiles. While executing the de-distortion matrix SW module 822, processor 810 can also execute a tile configuration submodule 823, a polynomial modeling submodule 824, and / or a non-polynomial modeling submodule 825. For example, processor 810 can execute tile configuration submodule 823 to configure the size of each CIS tile, at least in part, based on the memory bandwidth utilization of the LGDC system. Furthermore, processor 810 can execute polynomial modeling submodule 824 to determine the pixel mapping based on a polynomial lens model. Further still, processor 810 can execute non-polynomial modeling submodule 825 to determine the pixel mapping based on a non-polynomial lens model.

[0074] Processor 810 may also execute block LUT SW module 826 to generate multiple block LUTs that identify one or more DIS blocks associated with each de-distortion matrix. While executing block LUT SW module 826, processor 810 may also execute block configuration submodule 827 and / or block ordering submodule 828. For example, processor 810 may execute block configuration submodule 827 to configure the size of each DIS block based at least in part on the memory bandwidth utilization of the LGDC system. Furthermore, processor 810 may execute block ordering submodule 828 to determine the extraction order of each DIS block based at least in part on the order of the de-distortion matrix.

[0075] Figure 9 This is an illustrative flowchart depicting an example image processing operation 900 according to some embodiments. For example, refer to... Figure 7 Operation 900 can be performed by the block dedistortion circuit 700 to generate corrected image data based on the distorted image data.

[0076] The dedistorting circuit can retrieve multiple dedistorting matrices in an ordered sequence for remapping pixels of a first image from the distorted image space to the corrected image space (910). The first image may depict a scene in the distorted image space. For example, the dedistorting circuit can retrieve a stride remapping dataset from memory. The dataset may include multiple dedistorted pixels arranged in raster order. Each dedistorting matrix can define a pixel mapping between a corresponding CIS patch and one or more DIS patches.

[0077] The de-distorting circuitry can retrieve pixel blocks (920) of the first image based at least in part on the order of the de-distorting matrices. For example, the stride remapping dataset may also include block LUTs that identify the DIS blocks serving each de-distorting matrix. Therefore, the order of the DIS blocks may depend at least in part on the order in which the de-distorting circuitry retrieves and processes the de-distorting matrices. In some embodiments, the de-distorting circuitry may retrieve the DIS blocks in the order specified in the LUT.

[0078] The de-distortion circuit can generate multiple image patches based on pixel blocks of a first image, wherein each image patch in the image patches is interpolated from one or more pixel blocks in the retrieved pixel blocks using a corresponding de-distortion matrix (930). For example, the de-distortion circuit can process each patch in the corrected image space based on retrieved DIS blocks and corresponding de-distortion matrices. In some embodiments, the de-distortion circuit can generate each CIS patch based on bicubic interpolation of reference points indicated by the de-distortion matrix in the DIS block.

[0079] Then, the de-distorting circuit can combine multiple image tiles to produce a second image (940) depicting the scene in the corrected image space. (See above regarding...) Figure 3 When arranged in raster order, multiple CIS tiles can form a full-frame image in the corrected image space. In some embodiments, the de-distortion circuit can write each inset tile back to memory before processing the next tile (or the next de-distortion matrix) in the corrected image space. Therefore, when the de-distortion circuit completes de-distortion, the image rendering device (or image analysis device) can render the second image in near real-time.

[0080] Those skilled in the art will understand that information and signals can be represented using any of a variety of different processes and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the foregoing specification can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0081] Furthermore, those skilled in the art will understand that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in connection with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally according to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0082] The methods, sequences, or algorithms described in connection with the aspects disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated into the processor.

[0083] In the foregoing specification, embodiments have been described with reference to specific examples. However, it will be apparent that various modifications and changes can be made therein without departing from the broader scope of this disclosure as set forth in the appended claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. An image processing method, comprising: Multiple dedistortion matrices for remapping pixels of a first image from a distorted image space to a corrected image space are retrieved in an ordered sequence, the first image depicting a scene in the distorted image space; The pixel blocks of the first image are retrieved at least in part based on the order of the dedistortion matrices, wherein each of the plurality of dedistortion matrices defines a mapping between a corresponding image patch in the corrected image space and one or more pixel blocks in the distorted image space, and each image patch in the corrected image space comprises a plurality of pixels; A plurality of image patches are generated based on the pixel blocks of the first image, wherein each image patch is interpolated from one or more pixel blocks of the retrieved pixel blocks using a corresponding de-distortion matrix in the de-distortion matrix; as well as The plurality of image tiles are combined to generate a second image depicting the scene in the corrected image space; The ordered sequence is at least partially based on the raster order of the image patches in the second image; The retrieval of pixel blocks in the first image is based on a lookup table, which identifies the pixel blocks to be retrieved based at least on the de-distortion matrix.

2. The method according to claim 1, wherein each pixel block in the pixel block includes a plurality of first pixels of the first image, and wherein each image block in the image block includes a plurality of second pixels of the second image.

3. The method of claim 2, wherein the number of the plurality of first pixels in each pixel block is greater than the number of the plurality of second pixels in each image patch.

4. The method of claim 2, wherein each of the second pixels is interpolated from one or more first pixels in the first pixels using bicubic interpolation.

5. The method of claim 1, wherein the combination comprises: The plurality of image tiles are stored in the raster order.

6. The method of claim 1, wherein the second image is divided into a plurality of strides having uniform height and width, the height of each stride being equal to the height of one image patch in the image patch, and the width of each stride being equal to the width of the second image, and wherein each stride sequentially comprises two or more image patches in the image patch.

7. The method of claim 6, wherein the retrieval of the de-distortion matrix comprises: Retrieve the dedistortion matrix for the image patch used to generate the first of the plurality of strides; as well as Only after each of the dedistortion matrices associated with the first step is retrieved, the dedistortion matrix for generating the image patch of the plurality of steps is retrieved.

8. The method of claim 6, wherein the retrieval of the pixel block comprises: Retrieve a first lookup table, the first lookup table identifying the pixel block of the image patch used to generate the first of the plurality of strides; as well as Only after each pixel block identified by the first lookup table has been retrieved, a second lookup table is retrieved, the second lookup table identifying the pixel block of the image patch used to generate the second stride of the plurality of strides.

9. The method of claim 1, wherein the first image is captured by an image capturing device, the method further comprising: The plurality of dedistortion matrices are generated at least in part based on one or more geometric distortion parameters or focal lengths of the lens of the image capturing device.

10. The method of claim 9, wherein the lens introduces nonlinear distortion in the scene depicted by the first image, and wherein the scene depicted by the second image is linear.

11. The method of claim 9, further comprising: The size of the image patch or pixel block is selected at least in part based on one or more geometric distortion parameters or the focal length.

12. The method of claim 9, wherein the de-distortion matrix is ​​generated based on a polynomial lens model.

13. The method of claim 9, wherein the de-distortion matrix is ​​generated based on a non-polynomial lens model.

14. The method according to claim 1, further comprising: The size of the image patch or pixel block is selected at least in part based on the storage bandwidth requirements for generating each image patch in the image patch.

15. The method of claim 14, wherein the selection is further based on the image format or pixel depth of the first image.

16. A de-twisting circuit, comprising: Storage buffer; as well as The hardware logic is configured as follows: Multiple dedistortion matrices for remapping pixels of a first image from a distorted image space to a corrected image space are retrieved in an ordered sequence. The first image depicts a scene in the distorted image space. Each of the multiple dedistortion matrices defines a mapping between a corresponding image patch in the corrected image space and one or more pixel blocks in the distorted image space. Each image patch in the corrected image space includes multiple pixels. The pixel blocks of the first image are read into the storage buffer at least in part based on the order of the de-distortion matrix; Multiple image patches are generated based on the pixel blocks of the first image, wherein each image patch is interpolated from one or more pixel blocks of the retrieved pixel blocks using a corresponding de-distortion matrix in the de-distortion matrix; as well as The plurality of image tiles are written to an external memory to generate a second image depicting the scene in the corrected image space; The ordered sequence is at least partially based on the raster order of the image patches in the second image; The hardware logic is used to read the pixel block into the storage buffer by: retrieving a lookup table, which identifies the pixel block to be retrieved based at least on the de-distortion matrix.

17. The de-distortion circuit of claim 16, wherein each pixel block in the pixel block includes a plurality of first pixels of the first image, and wherein each image block in the image block includes a plurality of second pixels of the second image.

18. The de-distortion circuit of claim 17, wherein the plurality of first pixels in each pixel block are greater than the plurality of second pixels in each image patch.

19. The de-distortion circuit of claim 16, wherein the hardware logic is further configured to store the plurality of image tiles in the raster order.

20. The de-distortion circuit of claim 16, wherein the second image is divided into a plurality of steps having uniform height and width, the height of each step being equal to the height of one image patch in the image patch, and the width of each step being equal to the width of the second image, and wherein each step comprises two or more image patches in the image patch.

21. The de-twisting circuit of claim 16, wherein the hardware logic is configured to retrieve the de-twisting matrix by: Retrieve the dedistortion matrix for generating the first step of a plurality of steps; and Only after each of the dedistortion matrices associated with the first step is retrieved, the dedistortion matrix for generating the image patch of the plurality of steps is retrieved.

22. The de-twisting circuit of claim 16, wherein the hardware logic is configured to read the pixel block into the storage buffer by: Retrieve a first lookup table, the first lookup table identifying the pixel block of the image patch used to generate the first of a plurality of strides; and Only after each pixel block identified by the first lookup table has been retrieved, a second lookup table is retrieved, the second lookup table identifying the pixel block of the image patch used to generate the second stride of the plurality of strides.

23. A system comprising: processor; The memory stores instructions that, when executed by the processor, cause the system to: A plurality of dedistortion matrices are generated in an ordered sequence for remapping pixels from a distorted image space to a corrected image space. Each of the plurality of dedistortion matrices defines a mapping between a corresponding image patch in the corrected image space and one or more pixel blocks in the distorted image space, and each image patch in the corrected image space comprises a plurality of pixels; and The de-twisting circuit is configured as follows: The pixel blocks of a first image, which depicts a scene in the distorted image space, are retrieved at least in part based on the order of the dedistortion matrix. Multiple image patches are generated based on the pixel blocks of the first image, wherein each image patch is interpolated from one or more pixel blocks of the retrieved pixel blocks using a corresponding de-distortion matrix in the de-distortion matrix; as well as The plurality of image tiles are combined to generate a second image depicting the scene in the corrected image space; The ordered sequence is at least partially based on the raster order of the image patches in the second image; The retrieval of pixel blocks in the first image is based on a lookup table, which identifies the pixel blocks to be retrieved based at least on the de-distortion matrix.

24. The system of claim 23, wherein the de-distortion matrix is ​​generated based on a polynomial lens model.

25. The system of claim 23, wherein the de-distortion matrix is ​​generated based on a non-polynomial lens model.

26. The system of claim 23, wherein execution of the instructions further causes the system to: The size of the image patch or pixel block is selected at least in part based on the storage bandwidth requirements for generating each image patch in the image patch.

27. The system of claim 26, wherein the selection is further based on the image format or pixel depth of the first image.

Images