Image Dewarping Circuit

Abstract

A method and apparatus for image processing includes a dewarping circuit configured to obtain a plurality of dewarping matrices for remapping, in an ordered sequence, pixels of a first image depicting a scene in the distorted image space from the distorted image space to a corrected image space. The dewarping circuit reads blocks of pixels of the first image into a memory buffer based at least in part on the order of the dewarping matrices, and generates a plurality of image tiles based on the blocks of pixels of the first image. Each of the image tiles is interpolated from one or more obtained blocks of pixels using one or more dewarping matrices, respectively. The dewarping circuit writes the plurality of image tiles to an external memory to generate a second image depicting the scene in the corrected image space.

Description

[Technical field]

[0001] The present embodiments relate generally to image processing. [Background technology]

[0002] Image processing allows the captured image to be depicted on the display in such a way that the original scene can be reproduced as accurately as possible, taking into account the capabilities or limitations of the image capture device (such as a camera). For example, a wide-angle lens may be used to capture a larger field of view (FOV) than a standard rectilinear lens. However, the wide-angle lens also introduces geometric nonlinear distortions into the captured image. Examples of geometric distortions include radial or barrel distortions, and tangential distortions. Thus, image processing may correct the geometric distortions of the captured image so that the scene is shown more linearly. The process of reversing the geometric distortions introduced by the lens optics and / or the perspective of the image capture device is often called dewarping. Summary of the Invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] A method and apparatus for image processing is disclosed. One innovative aspect of the present disclosure may be implemented as a method of image processing. In some embodiments, the method may include obtaining a plurality of dewarping matrices for remapping pixels of a first image depicting a scene in a distorted image space from the distorted image space to a corrected image space in an ordered sequence, obtaining blocks of pixels of the first image based at least in part on an order of the dewarping matrices, generating a plurality of image tiles based on the blocks of pixels of the first image, where each of the image tiles is interpolated from the obtained one or more blocks of pixels using one or more respective dewarping matrices, and combining the plurality of image tiles to generate a second image depicting the scene in the corrected image space.

[0005] Another innovative aspect of this disclosure may be implemented as an image processing system. In some embodiments, the system may include a processor, a memory, and a dewarping circuit that, when executed by the processor, causes the system to generate, in an ordered sequence, a plurality of dewarping matrices for remapping pixels from a distorted image space to a corrected image space.

[0006] The dewarping circuitry is configured to obtain blocks of pixels of a first image illustrating a scene in a distorted image space based at least in part on an ordering of the dewarping matrices, generate a plurality of image tiles based on the blocks of pixels of the first image, where each of the image tiles is interpolated from the one or more obtained blocks of pixels using one or more dewarping matrices respectively, and combine the plurality of image tiles to generate a second image illustrating the scene in the corrected image space. [Brief description of the drawings]

[0007] The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0008] [Figure 1] FIG. 1 illustrates a block diagram of an image capture and display system according to some embodiments.

[0009] [Diagram 2] FIG. 2 illustrates a block diagram of a lens geometric distortion correction (LGDC) circuit according to some embodiments.

[0010] [Diagram 3] FIG. 3 illustrates an exemplary corrected image space according to some embodiments.

[0011] [Figure 4] FIG. 4 shows an exemplary pixel mapping between the corrected image space and the distorted image space.

[0012] [Diagram 5] FIG. 5 illustrates a block diagram of a LGDC system according to some embodiments.

[0013] [Figure 6] FIG. 6 illustrates a block diagram of an image space remapping controller according to some embodiments.

[0014] [Figure 7] FIG. 7 illustrates a block diagram of a tile dewarping circuit according to some embodiments.

[0015] [Figure 8] FIG. 8 illustrates another block diagram of an image space remapping controller according to some embodiments.

[0016] [Figure 9] FIG. 9 is an example flowchart illustrating an example image processing operation according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, to provide a thorough understanding of the present disclosure. The term "coupled" as used herein means directly connected or connected through one or more intervening components or circuits. Furthermore, in the following description, and for purposes of explanation, specific nomenclature is used to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the disclosure. Some portions of the following detailed description are presented in the form of processes, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. Interconnections between circuit elements or software blocks may be depicted as buses or single signal lines. Each of the buses may alternatively be a single signal line, and each of the single signal lines may alternatively be a bus, and a single signal line or bus may represent any one or more of a myriad of physical or logical mechanisms for communication between components.

[0018] As will become apparent from the discussion below, unless specifically stated otherwise, discussions made throughout this application using expressions such as "accessing," "receiving," "transmitting," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," and the like, will be recognized as referring to the actions and processing of a computer system (or similar electronic computing device) that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers, or other such information storage, transmitter, or display device.

[0019] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules or components may be implemented together in an integrated logic device or as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable non-transitory storage medium that includes instructions that, when executed, implement one or more of the methods described above. The computer-readable non-transitory storage medium may form part of a computer program product. In addition, the computer program product may include packaging materials.

[0020] The non-transitory processor-readable storage medium may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. The present techniques may additionally or alternatively be implemented, at least in part, by a processor-readable communications medium that conveys or communicates code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer or other processor.

[0021] The various example logic blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors. As used herein, the term "processor" may refer to any general-purpose processor, conventional processor, controller, microcontroller, special purpose processor, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0022] 1 is a block diagram of an image capture and display system 100 according to some embodiments. The system 100 includes an image capture device 110, an image processor 120, and an image display device 130. The image capture device 110 captures a pattern of light 101 from a scene and converts the captured light 101 into digital image capture data 102. The image capture data 102 may correspond to a digital image of the scene. The image display device 130 displays the digital image by reproducing the light pattern on a corresponding display surface. In some embodiments, the image capture device 110 may be a camera and the image display device 130 may be an electronic display (such as a television, computer monitor, smartphone, etc.).

[0023] The image processor 120 performs image processing on the image capture data 102 to generate image representation data 103 that can be used to more accurately reproduce the original scene (captured by the image capture device 110) on the image display device 130. In some embodiments, the image processor 120 may be integrated into or otherwise included in the image capture device 110. The image processor 120 includes an image signal processing (ISP) module 122 and a lens geometry distortion correction (LGDC) module 124. The ISP module 122 may correct various pixel distortions in the image capture data 102 to improve the quality of the digital image. Exemplary pixel distortions include, but are not limited to, vignetting, aberrations, and noise.

[0024] The LGDC module 124 may convert or transfer the image capture data 102 to another image space that is more suitable or optimized for display on the image display device 130. For example, a wide-angle lens of the image capture device may introduce geometric distortions (such as radial or barrel distortion, tangential distortion, keystone distortion, etc.) to the scene depicted in the image capture data 102. Thus, the LGDC module 124 may be configured to correct the geometric distortions introduced by the image capture device 110 so that the captured image or scene appears more rectilinear on the image display device 130. In some embodiments, the image representation data 103 may be provided to other devices in addition to or instead of the image display device 130. For example, computer vision processing may benefit from rectilinearized images. Examples of computer vision processing may include, but are not limited to, face detection, object detection, and object tracking.

[0025] Aspects of the present disclosure recognize that certain applications may require near real-time processing of image capture data 102. Exemplary real-time applications include, but are not limited to, streaming video applications (such as from content delivery networks, video surveillance systems, video conferencing systems, etc.) and computer vision applications (such as face recognition systems, automotive driver assistance systems, and augmented reality (AR) systems). However, the operation of LGDC requires complex calculations and data manipulation. A fully software dewarping solution may consume a significant portion of the bandwidth or processing power of a central processing unit (CPU). On the other hand, a fully hardware dewarping architecture may occupy a significant amount of area and may have limited flexibility to reconfigure or adjust the dewarping algorithm for different lens geometries.

[0026] In some embodiments, the image processor 120 may implement the LGDC module 124 as a combined hardware and software solution. More specifically, aspects of the present disclosure may leverage the flexibility and scalability of software (e.g., executed by a CPU or a special purpose processor) to generate instructions for remapping one or more pixels of the image capture data 102 to one or more pixels of the image representation data 103 based on parameters of a lens of the image capture device 110. Aspects of the present disclosure may further leverage the speed and bandwidth of hardware logic to perform the calculations and data manipulations specified by the instructions generated by the CPU.

[0027] In some embodiments, the image space (e.g., "corrected image space") of the image representation data 103 may be divided or subdivided into tiles so that the dewarping operation may be performed at a finer granularity. For example, the size of each tile may be configured or optimized based at least in part on the memory bandwidth of the LGDC module 124. By performing pixel remapping operations at the granularity of tiles, the amount of hardware overhead required to dewarp a frame or image of the image capture data 102 is significantly reduced. This may further reduce the latency or delay of image processing such that the image representation data 103 may be generated in near real-time.

[0028] 2 is a block diagram of a lens geometric distortion correction (LGDC) circuit 200, according to some embodiments. The LGDC circuit 200 may be one embodiment of the LGDC module 124 of FIG. 1. As such, the LGDC circuit 200 may be configured to correct geometric distortion in the distorted image 202. For example, the geometric distortion may be introduced by a lens and / or other characteristics of an image capture device used to acquire or otherwise generate the distorted image 202. In some embodiments, the dewarping circuitry 210 may be configured to remap one or more pixels of the distorted image 202 to one or more pixels of the corrected image 204 such that the corrected image 204 appears substantially rectilinear.

[0029] In the embodiment of FIG. 2, distorted image 202 may be captured by a camera (or other image capture device) with a wide-angle lens. As a result, the scene depicted in distorted image 202 appears radially distorted. For example, objects with straight lines (such as buildings) appear curved in distorted image 202. In contrast, the scene depicted in corrected image 204 appears linear. For example, objects with straight lines appear linear in corrected image 204. As used herein, the term "distorted image space, or DIS" refers to the image space of distorted image 202 (e.g., before dewarping) and the term "corrected image space, or CIS" refers to the image space of corrected image 204 (e.g., after dewarping).

[0030] The LGDC circuit 200 includes dewarping circuitry 210 and memory 220. The dewarping circuitry 210 may receive image data corresponding to the distorted image 202 and may output corrected image data corresponding to the corrected image 204. In some embodiments, the corrected image 204 may have the same dimensions (e.g., pixel height and pixel width) as the distorted image 202. However, the corrected image 204 may depict less of the scene than the distorted image 202. For example, a building in the upper left corner of the distorted image 202 is not depicted in the corrected image 204. The dewarping circuitry 210 may then interpolate each pixel of the corrected image 204 from one or more pixels of the distorted image 202.

[0031] In some embodiments, the dewarp circuitry 210 may determine the remapping of pixels from the distorted image space to the corrected image space based at least in part on one or more lens parameters 206 of the camera of the image capture device. Exemplary lens parameters 206 may include, but are not limited to, the lens geometry and focal length. Since the dewarping process is to reverse the geometric distortion introduced by the camera lens, the pixel remapping may be the inverse of the pixel mapping created by the camera lens. In some aspects, the dewarp circuitry 210 may determine the pixel remapping calculation based on a polynomial lens model. In some other aspects, the dewarp circuitry 210 may determine the pixel remapping calculation based on a non-polynomial lens model.

[0032] The memory 220 may be configured to store or buffer any data required to complete the process of dewarping. The data stored in the memory 220 may include, but is not limited to, the camera lens parameters 206, pixel remapping calculations (e.g., matrices), an index order for fetching blocks of the distorted image, image data for the distorted image 202, and image data for the corrected image 204. Because the memory 220 has a limited storage capacity, it may be inefficient (if not impossible) to store a complete set of pixel data for an entire frame of the distorted image 202 and / or the corrected image 204. Therefore, in some embodiments, the LGDC circuit 200 may be configured to generate a small portion (e.g., a tile) of the corrected image 204 at a time. Because each portion of the corrected image 204 is interpolated only on a relatively small subset (e.g., a block) of the pixels of the distorted image 202, aspects of the present disclosure may significantly reduce the memory bandwidth requirements of the memory 220.

[0033] FIG. 3 illustrates an example corrected image space 300 according to some embodiments. The corrected image space 300 may be one embodiment of the image space of the corrected image 204 of FIG. 2. As shown in FIG. 3, the corrected image space 300 is divided or subdivided into a number of tiles T1(1)-Tn(m) of equal or uniform size. Each tile may include a number of pixels of the corrected image space 300. In some embodiments, the size of each tile (e.g., pixel height and pixel width) may be based at least in part on the memory bandwidth of the LGDC circuit (such as memory 220 of FIG. 2). In some other embodiments, the size of each tile may be based at least in part on the pixel depth and / or color format of the corrected image (such as the corrected image 204 of FIG. 2).

[0034] Each column of tiles may form a respective stride. For example, stride S1 may include tiles T1(1)-T1(m), stride S2 may include tiles T2(1)-T2(m), stride S3 may include tiles T3(1)-T3(m), and so on. Thus, each of the strides S1-Sn may span the width of the corrected image space 300 (e.g., corresponding to the combined width of the pixels of tiles T1(1)-T1(m)) and may have a height equal to the height of one pixel of the tile. In some embodiments, the dewarping process may be performed on each of the tiles, one stride at a time, in raster order. For example, the dewarping circuitry may process each tile of stride S1 successively (e.g., T1(1), T1(2), ..., T1(m)) before processing each tile of stride S2 from left to right (e.g., T2(1), T2(2), ..., T2(m)).

[0035] The size and dimensions of each stride may be based, at least in part, on the raster order of the display updates and / or the granularity of each update. In the embodiment of FIG. 3, strides S1-Sn are shown to correspond with horizontal rows of the rectified image space 300, e.g., to support near real-time dewarping for horizontal display updates. In other embodiments, the strides may correspond with vertical columns of the rectified image space 300, e.g., to support near real-time dewarping for vertical display updates. For example, the dewarping circuitry may sequentially process an entire column of tiles from top to bottom (e.g., T1(1), T2(1), ..., Tn(1)) before proceeding to process the next column of tiles from top to bottom (e.g., T1(2), T2(2), ..., Tn(2)). Additionally, in some embodiments, multiple strides of the rectified image space 300 may be processed simultaneously (e.g., using multi-threading).

[0036] Aspects of the present disclosure recognize that the mapping of pixels between the corrected image space 300 and the distorted image space may be non-linear. With reference to the example of FIG. 2, geometric distortions may be more pronounced at the edges or periphery of the distorted image 202 compared to the center of the image. Thus, pixels at the center of the corrected image 204 may be mapped to pixels at the center of the distorted image 202. However, pixels at the corners and edges of the corrected image 204 may not be mapped to pixels at the respective corners and edges of the distorted image 202. Rather, pixels at the corners and edges of the corrected image 204 may be mapped to pixels closer to the center of the distorted image 202.

[0037] FIG. 4 illustrates an example pixel mapping 400 between a corrected image space 410 and a distorted image space 420. The corrected image space 410 may be an embodiment of the corrected image space 300 of FIG. 3. In the embodiment of FIG. 4, the distortion is relatively less noticeable in the center of the distorted image space 420. For example, a tile 412 in the center of the corrected image space 410 may be mapped to a block of pixels 422 in the center of the distorted image space 420. However, the distortion becomes more noticeable toward the edges of the distorted image space 420. For example, a tile 414, a tile 416, and a tile 418 in the upper left corner of the corrected image space 410 may each be mapped to the same block of pixels 424 in the distorted image space 420.

[0038] Thus, in some embodiments, a respective dewarp matrix (or set of matrices) may be generated for each tile of the corrected image space 410. Each dewarp matrix may define a pixel mapping between a particular tile of the corrected image space 410 and a corresponding block of pixels in the distorted image space 420. In some embodiments, the size of each block of pixels (e.g., pixel height and pixel width) in the distorted image space 420 may be based at least in part on the size of each tile of the corrected image space 410 and / or the memory bandwidth of the LGDC circuitry. In some aspects, the size of the blocks of the corrected image space 410 and the size of the tiles of the distorted image space 420 may be configured at least in part based on the tile granularity of the memory storage format. More specifically, the tile granularity may limit the size of the blocks and / or tiles depending on the image format and pixel depth of the image stored in the memory.

[0039] In some embodiments, the relationship between the size of the blocks in the corrected image space 410 and the size of the tiles in the distorted image space 420 may also vary in relation to one or more lens distortion parameters. For example, when the distorted image space 420 represents a substantially wider field of view than the corrected image space 410, the block size in the distorted image space 420 may be significantly larger than the tile size in the corrected image space 410. However, for less significant distortions, the size of the blocks in the distorted image space 420 may be closer to the size of the tiles in the corrected image space 410. In some aspects, the blocks in the distorted image space 420 may be larger (e.g., at least twice the size) than the tiles in the corrected image space 410. As described in more detail below, this may reduce the number and / or frequency of block read operations (e.g., from memory) required for processing each tile of the corrected image space 410.

[0040] 5 is a block diagram of an LGDC system 500 according to some embodiments. The LGDC system 500 may be one embodiment of the LGDC circuit 200 of FIG. 2. Thus, the LGDC system 500 may be configured to correct geometric distortions in the image capture data 501. The LGDC system 500 includes an image space remapping controller 510, a memory 520, and a tile dewarp circuit 530.

[0041] The image space remapping controller 510 may be configured to generate remapping data 502 used to remap pixels of the image capture data 501 from the distorted image space to the corrected image space. In some embodiments, the remapping data 502 may comprise a number of dewarping matrices and a number of lookup tables (LUTs) that identify blocks of pixels of the image capture data 501 to be used by the dewarping matrices. As described in relation to Figures 3 and 4, each dewarping matrix may define a pixel mapping between a respective tile of the corrected image space and one or more blocks of pixels of the distorted image space. Thus, the dewarping matrices may define a sequence of dewarping operations (e.g., calculations) to be performed by the dewarping circuitry 530.

[0042] The image space remapping controller 510 may generate a dewarping matrix based at least in part on lens configuration data 503 associated with an image capture device used to capture the image capture data 501. For example, the lens configuration data 503 may indicate one or more lens parameters (such as a lens geometric configuration and focal length) of the image capture device. In some embodiments, the lens configuration data 503 may be provided to the remapping controller 510 during the manufacturing stage of the device. In some other embodiments, the lens configuration data 503 and / or new lens configuration data may be provided to the remapping controller 510 at any time after the manufacturing of the LGDC system 500. Thus, the LGDC system 500 may be configured (and reconfigured) to dewarp images captured by various image capture devices.

[0043] The LUT may specify the order in which blocks of pixels of the image capture data 501 are acquired and processed by the dewarp circuitry 530. The order of the blocks specified by the LUT may be based, at least in part, on the order in which the dewarp matrices are to be performed by the dewarp circuitry 530. More specifically, the LUT may ensure that the dewarp circuitry 530 acquires the necessary blocks of pixels in the correct order to interpolate each tile of the corrected image space. In some embodiments (e.g., as described above in connection with FIG. 3), the dewarp matrices may be processed in raster order by the dewarp circuitry 530. Thus, the order of the blocks specified by the LUT may also depend on the raster order of the tiles in the corrected image space.

[0044] The memory 520 may be configured to store or buffer any data necessary to complete the dewarping process. In some embodiments, the memory 520 may be a dynamic random access memory (DRAM). The memory 520 may include a remapping data store 522, a distorted image space (DIS) image data store 524, and a corrected image space (CIS) image data store 526. The remapping data store 522 may store or buffer the remapping data 502 from the remapping controller. The DIS image data store 524 may store or buffer the image capture data 501. In some aspects, the DIS image data store 524 may be configured to store a full frame of pixel data describing a scene in a distorted image space. The CIS image data store 526 may store or buffer image description data 507 to be output to another device (such as the image display device 130 of FIG. 1) or electronic system. In some aspects, the CIS image data store 526 may be configured to store a full frame of pixel data describing a scene in a corrected image space.

[0045] The dewarp circuitry 530 may be configured to generate image representation data 507 based on the image capture data 501. For example, the dewarp circuitry 530 may interpolate each pixel of the image representation data 507 from one or more pixels of the image capture data 501 (e.g., using bicubic interpolation techniques). In some embodiments, the dewarp circuitry 530 may operate at a granularity of each tile of the corrected image space. More specifically, the dewarp circuitry 530 may process each tile of the corrected image space sequentially based on the order (e.g., raster order) of the dewarp matrices stored in the remapping data store 522.

[0046] The dewarp circuitry 530 may retrieve the remapping data 502 from the remapping data store 522 to determine a dewarp matrix corresponding to each tile of the corrected image space. The dewarp circuitry 530 may also determine blocks of pixels (e.g., of the image capture data 501) to be processed based on the LUT included in the remapping data 502. For example, the dewarp circuitry 530 may retrieve one or more DIS image blocks 504 from the DIS image data store 524 in an order specified by the LUT. The dewarp circuitry 530 may perform bicubic interpolation on the DIS image blocks 504 as defined by the dewarp matrix to generate one or more CIS image tiles 506. The dewarp circuitry 530 may then store the CIS image tiles 506 in the CIS image data store 526 as image depiction data 507. The CIS image tiles 506 may thus be combined (e.g., in raster order) to generate a full-frame image depicting the scene of the image capture data 501 in the corrected image space.

[0047] In some embodiments, the remapping controller 510 may be implemented in software. For example, the remapping controller 510 may include a set of instructions executed by a general-purpose processor or CPU. This provides greater scalability and flexibility in the design and / or configuration of the dewarping matrix. For example, an algorithm for inverting the geometric distortion of a camera lens may be configurable (and reconfigurable) by a user. Thus, aspects of the present disclosure may support dewarping matrices based on polynomial or non-polynomial lens models (e.g., to correct radial or tangential distortions). In some other aspects, the dewarping matrix may be configured or updated to support additional geometric distortion corrections, including (but not limited to) tilt, zoom, rotation, and focus changes.

[0048] In some embodiments, the dewarp circuitry 530 may be implemented in hardware. For example, the dewarp circuitry 530 may comprise a set of transistors or logic gates hardened to perform the calculations or mathematical operations defined by the dewarp matrix, such as bicubic interpolation or other pixel remapping functions. This provides greater speed and bandwidth in interpolating the image representation data 507. By processing the image representation data 507 at the granularity of the tile size, aspects of the present disclosure may further reduce the latency of the dewarp process while optimizing the use of available bandwidth and resources. As a result, frames of image representation data 507 may be generated in near real-time (e.g., such that the image representation data 507 is output by the LGDC system 500 at substantially the same rate as the image capture data 501 is received by the system 500).

[0049] 6 is a block diagram of an image space remapping controller 600 according to some embodiments. The remapping controller 600 may be an embodiment of the image space remapping controller 510 of FIG. 5. The remapping controller 600 may thus be configured to generate remapping data to be used to remap pixels of the image capture data 501 of FIG. 5 from a distorted image space to a corrected image space. In some embodiments, the remapping controller 600 may generate a stride remapping data set 604 for each stride 602 in the corrected image space (CIS). The remapping controller 600 comprises a dewarping matrix generator 610 and an image block LUT generator 620.

[0050] The dewarp matrix generator 610 is configured to generate a plurality of dewarp matrices M(1)-M(m) based at least in part on the lens configuration data 603 associated with the image capture device. Each dewarp matrix M1(1)-M(m) may be associated with a respective CIS matrix tile T(1)-T(m) of the current CIS matrix stride 602. The matrix elements of each CIS matrix tile T() may refer to row and column indices of CIS pixels. In some embodiments, the dewarp matrices M(1)-M(m) may define a pixel mapping between pixels of the CIS and pixels of the distorted image space (DIS). Thus, the matrix elements of each dewarp matrix M() may refer to row and column indices of DIS pixels.

[0051] In some embodiments, the matrix generator 610 may process each of the CIS matrix tiles T(1)-T(m) in raster order (e.g., as described above in connection with FIG. 3). In some aspects, each element of the dewarp matrix may include a high-precision pixel reference to a pixel in the referenced DIS block. When processing each pixel of a given CIS matrix tile, the associated dewarp matrix element also identifies one or more fractional coordinates for referencing a DIS pixel-in-pixel in the DIS block from which the pixel is to be interpolated.

[0052] The image block LUT generator 620 is configured to generate a respective block lookup table (LUT) for each CIS matrix stride 602. Each LUT may identify one or more DIS blocks associated with the CIS matrix stride 602. Because memory buffer space may be limited, the LUT may specify the order in which the DIS blocks should be retrieved for processing by the dewarping circuitry (such as the tile dewarping circuitry 530 of FIG. 5). More specifically, the order of the DIS blocks may match the order of the dewarping matrices M(1)-M(m) such that the required DIS blocks can be pre-fetched for processing of each of the CIS matrix tiles T(1)-T(m).

[0053] In some embodiments, the LUT generator 620 may determine the DIS blocks to be included in the LUT based at least in part on the pixel references included in each of the dewarp matrices M(1)-M(m). For example, the LUT generator 620 may determine a region in the warped image space bounded by the maximum and minimum pixel references in a particular dewarp matrix (or set of matrices). The LUT generator 620 may further identify one or more DIS blocks bounded by (or matching) the identified region and associate the DIS blocks with the dewarp matrices. For example, each DIS block in the LUT may be identified by the column and row offset address of the DIS block along with the number of the CIS matrix to which it is attached or other indicator.

[0054] The LUT may be appended to the dewarping matrices M(1)-M(m) and stored in system memory (such as memory 520 of FIG. 5) as a stride-remapped data set 604. In some embodiments, the LUT may be compressed (e.g., using run-length encoding) to reduce storage overhead of the data set 604. In some other embodiments, the remapping matrices M(1)-M(m) may also be compressed to further reduce storage overhead and / or reduce bandwidth congestion. The LUT generator 620 may process each stride 602 in the corrected image space sequentially (e.g., in raster order) for each frame of image capture data. In some aspects, the LUT generator 620 may output each of the stride-remapped data sets 604 as a continuous data stream.

[0055] 7 is a block diagram of a tile dewarp circuit 700 according to some embodiments. The dewarp circuit 700 may be an embodiment of the tile dewarp circuit 530 of FIG. 5. Thus, the dewarp circuit 700 may be configured to generate corrected image data based on distorted image data. In some embodiments, the dewarp circuit 700 may process each tile of the corrected image space individually (e.g., in raster scan order) by interpolating tile pixel data 708 from pixel data 706 of a corresponding block in the distorted image space.

[0056] The dewarp circuit 700 includes a memory interface 710, a stride processing controller 720, a distorted image space (DIS) input buffer 730, a corrected image space (CIS) output buffer 740, an interpolator 750, and a timing controller 760. The memory interface 710 provides an interface through which the dewarp circuit 700 can communicate with a system memory (such as memory 520 of FIG. 5). The memory interface 710 may include a stride remapping (SR) data read client 712, a DIS image block read client 714, and a CIS image block write client 716. The stride remapping data read client 712 may read the stride remapping (SR) data set 702 from memory in response to a stride read request 701. The DIS image block read client 714 may read block pixel data 706 from memory in response to a block read request 705. The CIS image block write client 716 may write the tile pixel data 708 to memory in response to the tile write request 707. In some aspects, the CIS image block write client 716 may write the tile pixel data 708 to memory in raster scan order.

[0057] The stride processing controller 720 may process the strides of the corrected image space. As described in relation to FIG. 3, the corrected image space may have a height of a given pixel and may include multiple strides spanning the width of the corrected image space. The stride processing controller 720 may output each stride read request 701 for processing each stride of the corrected image space. In response to the stride read request 701, the stride remapping data read client 712 returns the SR data set 702 to the stride processing controller 720. As described in relation to FIG. 6, the SR data set 702 may include a block LUT and multiple dewarping matrices M(1)-M(m). Each dewarping matrix may define pixel mapping between each tile of the corrected image space and one or more DIS blocks. The LUT may identify the DIS blocks that are subject to each dewarping matrix. The stride processing controller 720 may store the block LUTs and dewarping matrices in a LUT buffer 722 and a matrix buffer 724, respectively. In some embodiments, the matrix buffer 724 may be configured to store only a single dewarping matrix at a time.

[0058] The DIS input buffer 730 may read block information 703 from the LUT buffer 722 to determine which DIS blocks should be retrieved from memory. For example, the block information 703 may be parsed from the LUT and provided to the DIS input buffer 730 in the order in which the dewarp matrices should be processed. The block information 703 may include an index of one or more DIS blocks to be read from memory and a count value indicating the number of dewarp matrices that reference each DIS block. The DIS input buffer 730 may output a block read request 705 for each DIS block (or set of DIS blocks) having an index specified in the current block information 703. The DIS image block read client 714 returns block pixel data 706 of the requested DIS block (or DIS blocks) to the DIS input buffer 730. The DIS input buffer 730 may store or buffer the block pixel data 706 until the subsequent block information 703 is read from the LUT buffer 722.

[0059] The interpolator 750 may process each tile of the corrected image space based on the block pixel data 706 and the respective dewarp matrix 704. For example, the interpolator 750 may read each dewarp matrix 704 from the matrix buffer 724 in raster order. For each dewarp matrix 704, the interpolator 750 may use the buffered block pixel data 706 to generate a corresponding tile of the corrected image space according to the pixel mapping specified by the dewarp matrix 704. In some embodiments, the interpolator 750 may generate each tile based on a bicubic interpolation of the reference points indicated by the dewarp matrix in the block pixel data 706. For example, the interpolator 750 may interpolate each tile using four lunar (Y) pixels and four chroma (UV) pixels of the block pixel data 706 and a fractional offset indicated by the dewarp matrix 704.

[0060] The CIS output buffer 740 may buffer the output of the interpolator 750 (e.g., tile pixel data 708) until a completed tile is stored in the CIS output buffer 740. When the tile pixel data 708 in the CIS output buffer 740 represents a complete tile of the corrected image space, the CIS output buffer 740 may write the tile back to system memory. For example, the CIS output buffer 740 may output a tile write request 707 to the CIS image block write client 716 along with the tile pixel data 708. In some embodiments, each completed tile may be written to system memory in raster order before the interpolator 750 begins processing the next tile (or next dewarp matrix 704) of the corrected image space. In response, the image rendering device (or image analysis device) may render the tile pixel data 708 in near real-time as dewarping by the dewarping circuitry 700 is completed.

[0061] The timing controller 760 may generate timing signals 709 to control the timing of the stride controller 720, the DIS input buffer 730, and the CIS output buffer 740. More specifically, the timing signals 709 may be used to synchronize the operation of the stride controller 720, the DIS input buffer 730, and the CIS output buffer 740. For example, the timing signals 709 may ensure that the block LUT is first loaded into the LUT buffer 722 and that the DIS input buffer 730 retrieves the block pixel data 706 specified by the LUT before the stride controller 720 retrieves the dewarp matrix 704 associated with the block pixel data 706. Because multiple dewarp matrices 704 may depend on the same block pixel data 706, the DIS input buffer 730 does not need to retrieve new block pixel data 706 for each successive dewarp matrix 704. In some aspects, the DIS input buffer 730 may output a new block read request 705 when the number of dewarping matrices processed by the interpolator 750 reaches a count value associated with the current DIS block (or multiple DIS blocks) stored in the DIS input buffer 730.

[0062] 8 is another block diagram of an image space remapping controller 800 according to some embodiments. The remapping controller 800 may be implemented in an LGDC system (such as the LGDC system 500 of FIG. 5). Thus, the remapping controller 800 may be one embodiment of the image space remapping controller 600 of FIG. 6. The remapping controller 800 comprises a processor 810 and a memory 820.

[0063] The memory 820 may include an image pixel data store 821 configured to store image capture data and / or image rendering data. For example, the image representation data may correspond to an image of a scene in a distorted image space, and the image representation data may correspond to an image of a scene in a rectified image space. The memory 820 may include a computer readable non-transitory storage medium (e.g., one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, hard drive, etc.) capable of storing at least the following software (SW) modules: A dewarp matrix SW module 822 that generates a plurality of dewarp matrices representing pixel mapping between tiles of the corrected image space (CIS) and blocks of the distorted image space (DIS), the dewarp matrix SW module 822 further including: a tile configuration submodule 823 that configures a size for each CIS tile based at least in part on memory bandwidth utilization of the LGDC system; a polynomial modeling submodule 824 for determining a pixel mapping based on a polynomial lens model; and a non-polynomial modeling submodule 825 for determining pixel mapping based on a non-polynomial lens model; and A block LUTSW module 826 that generates a number of block LUTs that identify one or more DIS blocks associated with each dewarp matrix, the block LUTSW module 826 further including: a block configuration submodule 827 that configures the size of each DIS block based at least in part on the memory bandwidth usage of the LGDC system; and o A block order sub-module 828 that determines a fetch order for each of the DIS blocks based, at least in part, on the order of the dewarping matrices. Each software module contains instructions that, when executed by processor 810, cause remapping controller 800 to perform a corresponding function.

[0064] The processor 810 may be any suitable processor or processors capable of executing scripts or instructions of one or more software programs stored in the remapping controller 800. For example, the processor 810 may execute instructions 822-828 to generate a stride remapping dataset (such as dataset 604 of FIG. 6) for each stride in the corrected image space. In some embodiments, the processor 810 may be configured to operate simultaneously or in parallel (e.g., using multi-threading) on ​​multiple strides for improved latency. More specifically, software process latency may be reduced by increasing the number of threads run simultaneously by the processor 810.

[0065] The processor 810 may execute the dewarp matrix SW module 822 to generate a plurality of dewarp matrices representing pixel mapping between tiles of the CIS and blocks of the DIS. In executing the dewarp matrix SW module 822, the processor 810 may further execute the tile setting submodule 823, the polynomial modeling submodule 824, and / or the non-polynomial modeling submodule 825. For example, the processor 810 may execute the tile setting submodule 823 to set a size of each CIS tile based at least in part on the memory bandwidth usage of the LGDC system. Furthermore, the processor 810 may execute the polynomial modeling submodule 824 to determine pixel mapping based on a polynomial lens model. Furthermore, the processor 810 may execute the non-polynomial modeling submodule 825 to determine pixel mapping based on a non-polynomial lens model.

[0066] The processor 810 may also execute a block LUTSW module 826 to generate a number of block LUTs that identify one or more DIS blocks associated with each dewarp matrix. In executing the block LUTSW module 826, the processor 810 may further execute a block setting submodule 827 and / or a block order submodule 828. For example, the processor 810 may execute the block setting submodule 827 to set a size of each DIS block based at least in part on the memory bandwidth usage of the LGDC system. The processor 810 may also execute the block order submodule 828 to determine a fetch order for each of the DIS blocks based at least in part on the order of the dewarp matrices.

[0067] 9 is an exemplary flow chart illustrating an example image processing operation 900 according to some embodiments. With reference to the example of FIG. 7, the operation 900 may be performed by the tile dewarping circuit 700 to generate corrected image data based on distorted image data.

[0068] The dewarp circuitry may retrieve 910 a plurality of dewarp matrices for remapping pixels of a first image from a distorted image space to a corrected image space in an ordered sequence. The first image may represent a scene in the distorted image space. For example, the dewarp circuitry may retrieve a stride remapping dataset from a memory. The dataset may include a plurality of dewarped pixels arranged in raster order. Each dewarp matrix may define a respective pixel mapping between a CIS tile and one or more DIS blocks.

[0069] The dewarp circuitry may retrieve blocks of pixels of the first image based at least in part on the order of the dewarp matrices (920). For example, the stride remapping data set may further include a block LUT that identifies the DIS blocks subject to each dewarp matrix. Thus, the order of the DIS blocks may be based, at least in part, on the order in which the dewarp matrices are to be retrieved and processed by the dewarp circuitry. In some embodiments, the dewarp circuitry may retrieve the DIS blocks in the order specified in the LUT.

[0070] The dewarp circuitry may generate a plurality of image tiles based on blocks of pixels of the first image, where each image tile is interpolated from one or more retrieved pixel blocks using one or more respective dewarp matrices (930). For example, the dewarp circuitry may process each tile of the corrected image space based on the retrieved DIS block and the respective dewarp matrix. In some embodiments, the dewarp circuitry may generate each CIS tile based on bicubic interpolation of reference points indicated by the dewarp matrices in the DIS block.

[0071] The dewarping circuitry may then combine the image tiles to generate a second image showing the scene in the corrected image space (940). As described above in connection with FIG. 3, the CIS tiles, when arranged in raster order, may form a full frame image in the corrected image space. In some embodiments, the dewarping circuitry may write each interpolated tile back to memory before processing the next tile (or next dewarp matrix) in the corrected image space. Thus, the image rendering device (or image analysis device) may render the second image in near real time once the dewarping by the dewarping circuitry is complete.

[0072] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0073] Moreover, those skilled in the art will appreciate that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the aspects disclosed herein may be implemented as electrical hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints required for the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0074] The methods, procedures, or algorithms described in connection with the aspects disclosed herein may be embodied directly in hardware, in software modules executed by a processor, or in a combination of both. The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor.

[0075] In the foregoing specification, embodiments have been described with reference to specific examples thereof. It will be apparent, however, that various modifications and changes may be made thereto without departing from the broader scope of the disclosure, as set forth in the appended claims. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A method for image processing, comprising: obtaining, in an ordered sequence, a plurality of dewarping matrices for remapping pixels of a first image depicting a scene in the distorted image space from the distorted image space to a corrected image space; obtaining blocks of pixels of the first image based at least in part on the ordering of the dewarping matrix; generating a plurality of image tiles based on the blocks of pixels of the first image; combining the image tiles to generate a second image showing the scene in a rectified image space; and Including, the second image is divided into a number of strides of uniform height and width; the height of each stride is equal to the height of one of the image tiles; the width of each stride is equal to a width of the second image; each of said strides includes two or more of said image tiles in sequence; Obtaining the block of pixels of the first image obtaining a first look-up table (LUT) that identifies the blocks of pixels used to generate each of the image tiles of a first stride of the plurality of strides and that identifies an order in which the blocks of pixels used to generate the image tiles of the first stride should be obtained; obtaining the blocks of pixels used to generate the image tiles of the first stride based on the first LUT; Including, each of the image tiles is interpolated from the blocks of one or more pixels obtained using one or more of the dewarping matrices respectively; method.

2. each of the blocks of pixels includes a plurality of first pixels of the first image; each of the image tiles includes a plurality of second pixels of the second image; 2. The method of claim 1.

3. a number of the first pixels included in each block of pixels is greater than a number of the second pixels included in each image tile; The method of claim 2.

4. the ordered sequence being based at least in part on a raster order of the image tiles in the second image.

4. The method of any one of claims 1 to 3.

5. said combining including storing said plurality of image tiles in said raster order; The method of claim 4.

6. One of the blocks of pixels is permitted to be used in generating two or more image tiles of the plurality of image tiles.

6. The method of any one of claims 1 to 5.

7. Obtaining the dewarping matrix comprises: obtaining the dewarping matrix used to generate the image tiles of a first stride of the plurality of strides; obtaining the dewarping matrix used to generate the image tiles of a second stride of the plurality of strides only after each of the dewarping matrices corresponding to the first stride has been obtained; Including, The method of claim 6.

8. Obtaining the block of pixels further comprises: obtaining a second LUT that identifies the blocks of pixels used to generate each of the image tiles of a second stride of the plurality of strides only after each of the blocks of pixels identified by the first LUT has been obtained, and that specifies an order in which the blocks of pixels used to generate each of the image tiles of the second stride should be obtained; Including, 8. The method of claim 6 or 7.

9. A memory buffer; Hardware logic; Equipped with The hardware logic comprises: obtaining, in an ordered sequence, a number of dewarping matrices for remapping pixels of a first image depicting a scene in the distorted image space from the distorted image space to a corrected image space; reading blocks of pixels of the first image into the memory buffer based at least in part on the ordering of the dewarping matrix; generating a plurality of image tiles based on the blocks of pixels of the first image; writing the plurality of image tiles to an external memory to generate a second image showing the scene in the rectified image space; It is configured as follows: the second image is divided into a number of strides of uniform height and width; the height of each stride is equal to the height of one of the image tiles; the width of each stride is equal to a width of the second image; each of said strides includes two or more of said image tiles in sequence; Obtaining the block of pixels of the first image obtaining a first look-up table (LUT) that identifies the blocks of pixels used to generate each of the image tiles of a first stride of the plurality of strides and that identifies an order in which the blocks of pixels used to generate each of the image tiles of the first stride should be obtained; obtaining the block of pixels of the first image based on the first LUT; Including, each of the image tiles is interpolated from the blocks of one or more pixels obtained using one or more of the dewarping matrices respectively; Dewarp circuit.

10. It is a system, A processor; A memory for storing instructions; A dewarp circuit; Equipped with The instructions, when executed by the processor, cause the system to generate, in an ordered sequence, a plurality of dewarping matrices for remapping pixels from a distorted image space to a corrected image space; The dewarping circuit comprises: obtaining a block of pixels of a first image indicative of a scene in the distorted image space based at least in part on the ordering of the dewarping matrix; generating a plurality of image tiles based on the blocks of pixels of the first image; combining the plurality of image tiles to generate a second image showing the scene in the corrected image space. It is configured as follows: the second image is divided into a number of strides of uniform height and width; the height of each stride is equal to the height of one of the image tiles; the width of each stride is equal to a width of the second image; each of said strides includes two or more of said image tiles in sequence; Obtaining the block of pixels of the first image obtaining a first look-up table (LUT) that identifies the blocks of pixels used to generate each of the image tiles of a first stride of the plurality of strides and that identifies an order in which the blocks of pixels used to generate each of the image tiles of the first stride should be obtained; obtaining the block of pixels of the first image based on the first LUT; Including, each of the image tiles is interpolated from the blocks of one or more pixels obtained using one or more of the dewarping matrices respectively; system.

Images