Cache optimization warp engine

Abstract

This document describes systems and techniques for a cache optimization warp engine. In an aspect, a computing device having a first cache memory, a second cache memory, a system memory, and a cache optimization warp engine is configured to receive a warp input image including a plurality of pixel information. Following a coordinate sequence, the cache optimization warp engine scans the plurality of pixel information and loads a first portion of the pixel information into a first cache memory and a second portion of the pixel information into a second cache memory. Based on the first and second portions of the plurality of pixel information, the cache optimization warp engine determines the first and second portions of the plurality of pixel information of the corrected output image, respectively. The cache optimization warp engine stores a first portion and a second portion of pixel information of the corrected output image as an output image in a system memory.

Description

Background Art

[0001] Many computing devices may include cameras for taking photos and videos. Mobile computing devices (e.g., smartphones, tablet computers) are typically small in physical size and may not have room for large lenses or image sensors, which are important for collecting light to obtain better images or videos. Therefore, many mobile computing devices may include an image processing unit or central processing unit configured to provide computational photography features (e.g., light enhancement, blur reduction, distortion correction). Some mobile computing devices may include a warp engine that is configured to enhance images or videos via lens distortion correction, motion compensation, electronic image stabilization, rolling shutter correction, etc. Many warp engines may utilize cache memory to provide these enhancements.

[0002] Unfortunately, however, some warping engines may utilize cache memory in an unoptimized manner via tile-based image processing, which may include dividing the warped input image into tiles (e.g., portions of a grid) that are smaller than the warped input image. The warping engine may scan pixels within the tiles by following an unoptimized coordinate sequence. Scanning the pixels may include loading pixel information (e.g., position, color, gamma) associated with the pixels from a system memory (e.g., random access memory (RAM)) of a computing device. Portions of the warped input image may be loaded into the cache memory for use in processing subsequent pixels. However, by utilizing an unoptimized coordinate sequence, the cache memory may be larger than necessary, which increases the size of the computing device and reduces power efficiency. Summary of the invention

[0003] This document describes systems and techniques for a cache optimized warp engine (COWE). In aspects, a computing device having a first cache memory, a second cache memory, a system memory, and the COWE is configured to receive a warped input image including a plurality of pixel information. The COWE may receive the warped input image from a camera or a system memory of the computing device. The first cache memory may be a level 1 (L1) cache and include an access time that is 100 times faster than an access time of the system memory. The second cache memory may be a level 2 (L2) cache or a system level cache (SLC) and include an access time that is 10 times faster than an access time of the system memory.

[0004] Following the coordinate sequence, COWE may scan a plurality of pixel information and load a first portion of the pixel information into a first cache memory and a second portion of the pixel information into a second cache memory. COWE may scan a plurality of pixel information by loading a memory access unit (MAU) of the pixel information from a system memory into the first cache memory or the second cache memory. The first portion of the pixel information may include a MAU having pixel information that is more frequently accessed by COWE than the second portion of the pixel information stored in the second cache memory.

[0005] Based on the first and second parts of the plurality of pixel information, COWE may determine the first and second parts of the plurality of pixel information of the corrected output image, respectively. COWE may utilize a transformation model (e.g., translation, rotation) to first determine the positions of the first and second parts of the corrected output image. COWE may then interpolate the pixel information (e.g., color, gamma) of the first and second parts of the corrected output image using any of a variety of interpolation methods (e.g., bicubic interpolation).

[0006] COWE may store the first and second portions of the pixel information of the corrected output image as output images in a system memory. The first portion of the pixel information of the corrected output image may also include a subsequent first portion of the corrected output image determined by COWE based on the subsequent first portion of the distorted input image. Similarly, the second portion of the pixel information of the corrected output image may also include a subsequent second portion of the corrected output image determined by COWE based on the subsequent second portion of the distorted input image.

[0007] The details of one or more implementations are described in the accompanying drawings and in the following embodiments. Other features and advantages will become apparent from the embodiments, drawings, and claims. This summary is provided to introduce the subject matter further described in the embodiments. Therefore, the reader should not consider this summary to describe essential features or to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Details of one or more aspects of the cache optimized warp engine are described in this document with reference to the following drawings. Throughout the drawings, the same reference numerals may be used in different instances to indicate similar features or components. Figure 1 An example environment showing a computing device having a first cache memory, a second cache memory, a system memory, and a cache-optimized warping engine configured to correct a distorted input image; Figure 2 Show Figure 1 An example implementation of a computing device configured to provide a cache optimized warp engine; Figure 3 Describes an example approach that a cache-optimized warp engine can implement; Figure 4 shows an example of a corrected output image that can be divided into rows by a cache-optimized warp engine; Figure 5 shows an example of a warped input image that a cache-optimized warping engine can process into a corrected output image; Figure 6 An example of a cache optimized warp engine that loads pixel information from a system memory of a computing device is shown; Figure 7 illustrates various examples of memory access units that a cache optimized warp engine may obtain from a system memory of a computing device; Figure 8 An example of a memory access unit showing a cache optimized warp engine that can be loaded into a cache memory of a computing device; and Fig. 9 Rendering Cache Optimized Warping Engine Example methods that may be implemented on a warped input image to determine a corrected output image. DETAILED DESCRIPTION Overview

[0009] Computing devices (e.g., smart phones, tablet computers, computers) typically include cameras for capturing images and videos. Larger computing devices (e.g., dedicated cameras) may include large lenses and large image sensors that can capture more light, which is important for capturing higher quality images and videos. Smaller computing devices (e.g., smart phones) may not have enough physical space to include large lenses or image sensors, and therefore computational photography may be utilized. In aspects, computational photography may refer to digital image or video capture and processing techniques that use digital computing rather than optical processes. These computing devices may utilize various onboard components (e.g., cameras, processors) and utilize additional sensor data to digitally compute images. As an example, a computing device may include at least one cache memory, system memory (e.g., dynamic random access memory (DRAM)), and a distortion engine configured to process a distorted (e.g., distorted) input image into a corrected (e.g., undistorted) output image. The warping engine may access a warped input image from a system memory of the computing device, load portions of the warped input image into a cache memory, and process the warped input image into a corrected output image using portions of the warped input image in the cache memory and other portions of the warped input image in the system memory. The portions of the warped input image in the cache memory may be accessed faster than other portions of the system memory, and thus the warped input image may be processed into a corrected output image faster.

[0010] However, cache memory can be expensive in terms of physical space and power consumption. Therefore, a warp engine that is not optimized to utilize as little cache as possible can produce a physically larger and inefficient computing device. This can be particularly problematic in a mobile computing device that can be powered by a battery. For example, some warp engines can process a warp input image by dividing the warp input image into tile lines that include multiple tiles (e.g., portions of an image). The warp engine can process the warp input image tile by tile by scanning pixels within the tile along a scan line that travels from left to right, wherein the tile includes multiple scan lines from top to bottom. The warp engine can scan pixels by accessing pixel information associated with the pixels of the scan line from the system memory of the computing device and loading one or more scan lines of pixel information into the cache memory. The pixel information can be a number of bits (e.g., four bits, eight bits) and include position information, color information, and / or gamma information for a given pixel. The warp engine can interpolate the pixel information to produce a corrected output pixel.

[0011] For inter-tile pixels (e.g., pixels at the boundary between horizontally adjacent tiles), the same pixel information may be used to process the rightmost pixel of the current tile and the leftmost pixel of the next tile located to the right of the current tile. For example, the warping engine may load pixel information associated with the rightmost pixel of the current tile from the system memory of the computing device into a first cache memory of the computing device. The pixel information may include the position, color, and gamma information of the rightmost pixel and nearby pixels (e.g., two, three, or four pixels in each of the four cardinal directions) of the current tile. The warping engine may use an arbitrary transformation function (e.g., rotation, translation, stretching) to determine the position of the rightmost corrected output pixel of the corrected output tile of the corrected output image. After determining the position of the rightmost corrected output pixel, the warping engine may interpolate (e.g., using linear interpolation) the color and gamma information to determine the color and gamma information associated with the rightmost corrected output pixel of the corrected output tile of the corrected output image.

[0012] When the warp engine processes the leftmost pixel of the next tile, the warp engine may access the pixel information from the first cache memory. Using the pixel information in the first cache memory, the warp engine may determine (e.g., using bicubic interpolation) the pixel information (e.g., color, gamma) of the leftmost corrected output pixel of the next output tile of the corrected output image. Unfortunately, however, the warp engine may not access the position information of the leftmost pixel of the next tile before loading the scan lines of pixel information, so in some implementations, the warp engine may store one or more scan lines of pixel information in the first cache memory. Therefore, the first cache memory must be at least as large (e.g., in bits, bytes) as the one or more scan lines of pixel information.

[0013] Alternatively, a cache optimized warp engine (COWE) may process pixel information using rows (e.g., tile lines) rather than tiles. A row may include a portion of a warped input image that is as wide as the warped input image (e.g., in number of pixels) and as high as an integer division (e.g., one-tenth) of the warped input image. In other words, COWE may process the warped input image row by row by following an intra-row coordinate sequence scanning pixels from bottom to top and from left to right within the row. Scanning the pixels may include acquiring pixel information (e.g., color, gamma) stored in a system memory or cache of a computing device (e.g., if previously acquired). COWE may then follow the intra-row coordinate sequence for subsequent rows (e.g., row below, row above).

[0014] This document describes systems and techniques for COWE. The disclosed systems and techniques can address the shortcomings of caches that are not optimized for warp engines, which can increase the size of mobile computing devices and reduce the battery life of mobile computing devices. The conflicts between these shortcomings can be resolved by the disclosed systems and techniques, which can provide COWE, reduce cache size, and improve the battery life of mobile computing devices.

[0015] The following discussion describes an operating environment, techniques that may be employed in the operating environment, example apparatus, and example methods. Although systems and techniques for a cache-optimized warp engine are described, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, specific features and methods are disclosed as example implementations and are mentioned only for the sake of example. Example operating environment

[0016] Figure 1 An example environment 100 of a computing device 102 is shown that includes a first cache memory 104, a second cache memory 106, a system memory 108, a processor 110, and a display 112. The first cache memory 104 may be a level 1 (L1) cache associated with the processor 110. The second cache memory 106 may be a level 2 (L2) cache associated with the processor 110. Alternatively, at least one of the first cache memory 104 or the second cache memory 106 may be a system level cache (SLC) within the processor 110, a motherboard (not shown), or another suitable printed circuit board (PCB) within the computing device 102. The SLC may be a shared cache between the processor 110 and various peripheral devices or components of the computing device 102 (e.g., a display, a graphics processing unit (GPU), an audio codec).

[0017] The system memory 108 may be implemented as any of a variety of volatile memories, including random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), or the like. Alternatively or additionally, the system memory 108 may be implemented as any of a variety of non-volatile memories, including flash memory (e.g., solid-state drive (SSD)), read-only memory (ROM), magnetic computer storage devices (e.g., hard disk drive (HDD), floppy disk, tape), optical disks (e.g., compact disk (CD), digital video disk (DVD)), etc. The system memory 108 may be operatively coupled (e.g., electrically, physically, optically) to one or more components of the computing device 102.

[0018] The processor 110 may be implemented as any of a variety of single-core or multi-core processors, including a central processing unit (CPU), a graphics processing unit (GPU), an arithmetic logic unit (ALU), a reduced instruction set computer (RISC) microprocessor, an advanced RISC machine (ARM) microprocessor, etc. The processor 110 may utilize the first cache memory 104, the second cache memory 106, and the system memory 108 to provide some or all of the features described herein. The processor 110 may do so by executing computer-readable instructions stored on the system memory 108.

[0019] Display 112 may be implemented as any of a variety of displays, including a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED) display, a twisted nematic (TN) display, an in-plane switching (IPS) display, and the like. In addition, display 112 may be a display module including a layer stack including a display, a touch screen, a digitizer, a display driver integrated circuit (DDIC), a protective cover layer (e.g., glass, plastic), and the like. Display 112 may be included in a housing of computing device 102, or may be completely separate from computing device 102. Display 112 may be used to present content (e.g., images, video) and / or a graphical user interface (GUI) to a user of computing device 102.

[0020] Figure 1The computing device 102 is also shown to include a cache optimized warp engine 114 (COWE 114). COWE 114 may be implemented as a separate peripheral device or component (e.g., an image processing unit (IPU)) with a dedicated cache memory, or alternatively and as described above, the processor 110 may execute computer readable instructions stored on the system memory 108 to provide COWE 114 (e.g., as a software component). COWE 114 may utilize the first cache memory 104, the second cache memory 106, the system memory 108, and the processor 110 to process the warped input image into a corrected output image.

[0021] The input image may be distorted due to any of a variety of reasons, including lens distortion (e.g., barrel distortion, pincushion distortion), rolling shutter correction, geometric transformation (e.g., translation, rotation, Euclidean, affine, projection), motion compensation, electronic image stabilization (EIS), etc. COWE 114 can process the input image distorted due to lens distortion into a corrected output image by utilizing a predetermined transformation function based on the specific shape and material of the camera lens. For example, a manufacturer of a computing device with a camera can take average or median measurements of the curvature and material composition of a sample of lenses to predetermine the transformation function. COWE 114 can similarly process the input image distorted due to rolling shutter correction into a corrected output image. COWE 114 can process the input image distorted due to geometric transformation, motion compensation, or EIS by utilizing sensor data (e.g., accelerometer data). COWE 114 can determine an arbitrary transformation function based on the sensor data to process the distorted input image into a corrected output image.

[0022] Figure 1 Also shown is a user 116 of the computing device 102 having the COWE 114. In the example environment 100, the user 116 wishes to capture a photo of a scene including trees in the foreground and mountains in the background. As shown in display 112-1, in this example the scene is rotationally distorted and may be referred to as a distorted input image. Figure 1 1, a rotational distortion is shown by display 112-1, but the distorted input image may be distorted by any number of image distortions, including barrel distortion, pincushion distortion, perspective distortion, skew, curved horizontal distortion, panoramic distortion, combined distortions (e.g., complex distortion), etc. Particular image distortions may be caused by the lens of the camera, unstable capture of images or video, etc. Display 112-1 also shows a shutter button 118, which may be presented to user 116 by the GUI of computing device 102. User 116 may tap shutter button 118 to capture an image of the scene.

[0023] Display 112-2 shows that a distorted input image of a scene is corrected into a corrected output image by COWE 114 of computing device 102. COWE 114 may correct the distorted input image in response to user 116 tapping shutter button 118. COWE 114 may utilize first cache memory 104, second cache memory 106, and system memory 108 to correct the distorted input image.

[0024] As an example, COWE 114 may receive a distorted input image resulting from user 116 tapping shutter button 118 to capture a scene. When tapping shutter button 118, user 116 may have been unstable, causing the scene to rotate and distort into the distorted input image shown by display 112-1. The distorted input image may include multiple pixel information (e.g., color, position, gamma). COWE 114 may divide the distorted input image into a number of rows (e.g., two, five, 10, 20) that are as wide as the distorted input image and as high as an integer division of the distorted input image. For example, the distorted input image may be 1,920 pixels wide and 1,080 pixels high. In this example, each of the number of rows is 1,920 pixels wide and an integer division of 1,080 pixels high. The integer may be two, four, eight, 16, 32, etc., so that each of that number of rows is 540 pixels, 270 pixels, 135 pixels, 68 pixels, or 34 pixels high, respectively.

[0025] After dividing the warped input image into the number of rows, COWE 114 may scan a plurality of pixel information of the warped input image following a coordinate sequence including the number of rows. The coordinate sequence may include an intra-row coordinate sequence that proceeds from bottom to top and from left to right within the row and an inter-row coordinate sequence that proceeds from the topmost row to the bottommost row in a row-by-row manner. COWE 114 may scan the pixel information by accessing the pixel information from the system memory 108 in increments of a memory access unit (MAU).

[0026] The MAU may be any of a suitable size, including a number of pixels high (e.g., four, eight, 16, 32, 60) and another number of pixels wide (e.g., four, six, 10, 12), and may be measured in bits or bytes (B) (e.g., increments of eight bits). The specific size of the MAU accessed by COWE 114 may depend on the specific information included in the plurality of pixel information of the warped input image and the format of the system memory 108. The specific information may include color depth (e.g., 8-bit color, 128-bit color, 256-bit color), gamma, and position information. Additionally or alternatively, the pixel information may be compressed (e.g., reducing the number of bytes of the MAU) or uncompressed (e.g., increasing the number of bytes of the MAU). The format of the system memory 108 may include bus width (e.g., eight bits, 16 bits, 32 bits) and clock speed.

[0027] As a specific example, using a frame bandwidth reduced (FBR) format, a MAU may be 256 B and include 256 8-bit (e.g., 1 B) pixels. In another example, not using an FBR format, a MAU may still be 256 B but include only 32 64-bit (e.g., 8 B) pixels. As yet another example, a pixel may be described by 30 bits within a 32-bit container, and the associated MAU may include 48 pixels, for a total of 192 B of pixel information.

[0028] Following the coordinate sequence mentioned above, COWE 114 may load portions of the plurality of pixel information of the distorted input image into the first cache memory 104, the second cache memory 106, or both. COWE 114 may do so because portions of the plurality of pixel information may be referenced multiple times to process the pixels of the distorted input image into the corrected pixels of the corrected output image. In addition, accessing pixel information from the first cache memory 104 or the second cache memory 106 may be significantly faster than accessing pixel information from the system memory 108. For example, accessing pixel information from the second cache memory 106 may be 25 times faster than accessing pixel information from the system memory 108. Similarly, accessing pixel information from the first cache memory 104 may be 100 times faster than accessing pixel information from the system memory 108.

[0029] Specifically, COWE 114 may load a first portion of the intra-row pixel information into the first cache memory 104 and a second portion of the inter-row pixel information into the second cache memory 106. The first portion of the pixel information may include a number of scan lines from bottom to top (e.g., according to the coordinate sequence mentioned above) consisting of one or more MAUs, the number depending on the height of the row. The second portion of the pixel information may include a number of MAUs that form a boundary between the top row and the adjacent row below the top row. The first portion of the pixel information may be most frequently referenced by COWE 114 to determine the corrected output pixel, thus justifying why the first portion may be stored in the first cache memory 104 because the first cache memory is faster than the second cache memory 106 and the system memory 108. Similarly, the second portion of the pixel information may be referenced more frequently than the portion in the system memory 108, but not as frequently as the first portion in the first cache memory 104.

[0030] COWE 114 may determine a first corrected portion of multiple pixel information of a corrected output image shown by display 112-2 based on the first portion of the pixel information in the first cache memory 104. COWE 114 may do so by determining a transformation function from a distorted input pixel position to a corresponding corrected output pixel position. The transformation function may include a predetermined function based on the curvature of a lens of a camera that captured the distorted input image. Additionally or alternatively, the transformation function may be based on sensor data (e.g., accelerometer) captured with the distorted input image. COWE 114 may then use an interpolation method (e.g., linear interpolation, bilinear interpolation, cubic interpolation) on the first portion of the pixel information in the first cache memory 104 to determine a first corrected portion (e.g., color, gamma) of multiple pixel information of the corrected output pixel position.

[0031] COWE 114 may determine a second corrected portion of the plurality of pixel information of the corrected output image based on the second portion of the pixel information in the second cache memory 106. COWE 114 may similarly do so by determining a transformation function from the distorted input pixel positions in the second portion of the pixel information to the corresponding corrected output pixel positions. By performing the described method, COWE 114 may determine a remaining corrected portion of the plurality of pixel information of the corrected output image. COWE 114 may then store the corrected output image in the system memory 108 of the computing device 102.

[0032] By following the coordinate sequence mentioned above, the size of the first cache memory 104 and the second cache memory 106 can be optimized. Therefore, the computing device 102 can benefit from power saving and size reduction while correcting the distorted input image to the corrected output image. These benefits enable the computing device 102 to be physically smaller and consume less energy, thereby improving the user experience of the user 116. Example Device

[0033] In more detail, Figure 2 Show Figure 1An example implementation 200 of a computing device 102 configured to provide COWE 114. The computing device 102 is shown as various example devices. As non-limiting examples, the computing device 102 may be a smartphone 202a, a tablet computer 202b, a laptop computer 202c, a desktop computer 202d, a smart watch 202e, a pair of smart glasses 202f, a game controller 202g, a smart home speaker 202h, or a microwave appliance 202i. Although not shown, the computing device 102 may also be implemented as a health monitoring device, a personal media device, a drone, a household appliance, a security system or its device, a digital photo frame, etc. The computing device 102 may be wearable, non-wearable but mobile, or relatively fixed. In addition, the computing device 102 may be used with or embedded in many computing devices or peripheral devices (e.g., a vehicle, a personal computer). The computing device 102 may also include Figure 2 Additional interfaces or components omitted from .

[0034] Figure 2 The computing device 102 is shown to include reference Figure 1 Various components are depicted, including a first cache memory 104 , a second cache memory 106 , a system memory 108 , a processor 110 , a display 112 , and a COWE 114 . Figure 2 The computing device 102 is also shown to include a computer-readable medium 202 (CRM 202), which may include a memory medium 204 and a storage medium 206. The memory medium 204 may include one or more non-transitory storage devices, including RAM or DRAM. The storage medium 206 may include one or more temporary storage devices, including an SSD or a magnetic rotational HDD. The CRM 202 may also include an operating system 208 (OS 208) and applications 210, which may be stored as computer-readable instructions on the CRM 202. The processor 110 may execute the computer-readable instructions on the CRM 202 to provide some or all of the functions described herein. Although shown as a separate component, the system memory 108 may be included in the CRM 202 as a separate memory component, part of the memory medium 204, part of the storage medium 206, or part of both.

[0035] Figure 2The computing device 102 is also shown to include one or more sensors 212. The sensors 212 may include image sensors, audio sensors (e.g., microphones), accelerometers, barometers, ambient light sensors, thermometers, etc. One or more of the sensors 212 (e.g., accelerometers) may be utilized by the COWE 114 to determine the transformation function. Additionally or alternatively, the COWE 114 may receive the warped input image from one or more of the sensors 212 (e.g., image sensors, cameras). For example, the computing device 102 may be a smartphone 202a, which may include a camera or camera system (e.g., image sensor and lens).

[0036] In an implementation, COWE 114 may include one or more integrated circuits (ICs) (e.g., a power management integrated circuit (PMIC)), a system on a chip (SOC), a secure key storage device, hardware with embedded firmware, a PCB with various hardware components (e.g., a motherboard, a daughterboard), or any combination thereof. As described herein, COWE 114 may include one or more components of computing device 102, such as Figure 1 and Figure 2 As shown, the one or more components are configured to determine a corrected output image based on the distorted input image. In other implementations, the COWE 114 may be implemented as the computing device 102.

[0037] Although not shown, computing device 102 may also include input / output (I / O) ports, a system bus, an interconnect, or another data transmission system coupled to various components of computing device 102 or within computing device 102. As an example, I / O ports may enable computing device 102 to interact with other devices or users via peripheral devices, transmit any combination of digital signals and / or analog signals via wired means (e.g., Ethernet) or wireless means (e.g., radio). I / O ports may include any combination of internal ports or external ports, including universal serial bus ports, audio ports, video ports, etc. Various peripheral devices (e.g., manual input devices, external CRMs, speakers, displays) may be coupled to I / O ports.

[0038] Figure 3 Describe COWE (e.g. Figure 1 and / or Figure 2 Example method 300 that can be implemented by COWE 114 of FIG. 114. Example method 300 is shown as a flowchart with various shapes representing method steps or data (e.g., warping input images, pixel information). COWE is similar to Figure 1 and Figure 2 114 in, except as described in detail below. Thus, a COWE implementing the example method 300 may be operatively coupled to a first cache memory (e.g., Figure 1 and / or Figure 2 A first cache memory 104), a second cache memory (eg, Figure 1 and / or Figure 2 The second cache memory 106) and / or the system memory (e.g., Figure 1 and / or Figure 2 System memory 108, Figure 2 Alternatively or additionally, the COWE implementing the example method 300 may be a computing device (e.g., Figure 1 and / or Figure 2 A component of a computing device 102) or implemented as a computing device.

[0039] As shown, Figure 3 Example method 300 includes a transformation model 302 and a distortion grid buffer 304. Transformation model 302 may be based on a set of predetermined transformations associated with one or more camera lenses of a computing device (e.g., computing device 102). For example, a COWE for a computing device having a telephoto lens and a macro photography lens may include a transformation associated with the telephoto lens and a transformation associated with the macro photography lens. The transformations may be predetermined based on the curvature of the lens and configured to correct the associated lens distortion. Alternatively or additionally, transformation model 302 may be based on accelerometer data. For example, a user of a computing device (e.g., computing device 102) having a camera, an accelerometer, and a COWE may include a transformation associated with the telephoto lens and a transformation associated with the macro photography lens. Figure 1 A user 116) may take a photo of a scene. The user may be unsteady when capturing the photo, which may cause the photo to be skewed (e.g., tilted) or blurred. However, the accelerometer may record data that COWE can use to correct the skewed or blurred photo.

[0040] The distortion grid buffer 304 may be based on the transformation model 302 or a downsampled distortion value on a sparse grid. The sparse grid may include neighboring pixels of the current pixel of the distorted input image. COWE may utilize the distortion grid buffer 304 to map the position (e.g., x-coordinate and y-coordinate) of the current pixel of the distorted input image to the position of a pixel of the corrected output image.

[0041] COWE executing the example method 300 may use a transform generator 306 to generate a transform. As shown, COWE may utilize a transform model 302 and a distortion grid buffer 304 as inputs to the transform generator 306. Likewise, the generated transform may be based on a predetermined transform associated with the curvature of a camera lens or accelerometer data. The generated transform may be used by COWE to map the locations of pixels of the distorted input image 310 to the locations of pixels of the corrected output image 316. The transform generated by the transform generator 306 may be input to an address generator 308 of COWE.

[0042] The address generator 308 may utilize the generated transform to determine a mapping between locations of pixels of the warped input image 310 and locations of pixels of the corrected output image 316. The address generator 308 may also utilize the generated transform to determine a mapping between neighboring pixels of pixels of the warped input 310 image and neighboring pixels of pixels of the corrected output image 316. The generated transform and locations and associated pixel information (e.g., color, gamma) may be input to the pixel data interpolation 312.

[0043] COWE executing the example method 300 may utilize pixel data interpolation 312 to interpolate pixel information for pixels of the corrected output image 316. For example, COWE may utilize any of a variety of suitable interpolation methods to determine color and gamma information for pixels of the corrected output image 316. Some interpolation methods include piecewise constant interpolation, linear interpolation, bilinear interpolation, polynomial interpolation, spline interpolation, analog interpolation, cubic interpolation, bicubic interpolation, etc. The interpolation method may be based on the desired quality of the corrected output image 316 or the computing device having COWE (e.g., Figure 1 and / or Figure 2 The specific interpolation method is selected by COWE execution method 300 based on the computing power of the computing device 102).

[0044] like Figure 3 As shown, COWE performing the example method 300 may provide pixel information from the pixel data interpolation 312 as input data to an output crop 314. At the output crop 314, COWE may crop the corrected output image 316 by cropping portions of the corrected output image 316. COWE may crop an upper portion, a lower portion, a left portion, or a right portion of the corrected output image 316 at the output crop 314. COWE may determine which portions of the corrected output image 316 to crop at the output crop 314 based on a desired aspect ratio of the corrected output image 316, a data size, a data compression method, and the like.

[0045] Figure 4 It is shown that COWE (e.g. Figure 1 and / or Figure 2 COWE 114) is divided into rows (e.g., as referenced Figure 1 The corrected output image 400 (described) (e.g., Figure 3316). As shown, the corrected output image 400 may be W pixels wide and H pixels high. W and H may be any positive integers and depend on the desired format, data size, compression method, etc. of the corrected output image 400. For example, W may be 1,920 pixels and H may be 1,080 pixels (e.g., a 1080p image). As another example, W may be 3,840 pixels and H may be 2,160 pixels (e.g., a 4k image).

[0046] Figure 4 Also shown is the corrected output image 316 divided into rows 402. Rows 402 are as wide as the image (e.g., W pixels wide) and are m pixels high, where m is less than H and is an integer division of H. For example, H may be 1,080 pixels high and m may be 108 pixels high (e.g., an integer division of 10). As another example, H may be 2,160 pixels high and m may be 216 pixels high (e.g., an integer division of 10). As an additional example, H may be the same 2,160 pixels high and m may be 108 pixels high (e.g., an integer division of 20). The particular integer division may be based on the desired configuration of COWE for determining the corrected output image 316 or the computing capabilities of the computing device having COWE. As Figure 4 As shown in , the corrected output image 400 is divided into N rows, where N is a positive integer and equals H divided by m. For example, H may be 1,080 pixels and m may be 108 pixels, so that N is equal to 10.

[0047] Figure 5 It is shown that COWE can be processed into a corrected output image (e.g., Figure 3 and / or Figure 4 The distorted input image 500 (e.g., the corrected output image 316) Figure 3 Example of a distorted input image 310). Figure 5The distorted input image 500 is shown as four distorted rows 502. Although four distorted rows 502 are described, the number of rows may be any positive integer. The distorted rows 502 are shown as a first distorted row 502-1, a second distorted row 502-2, a third distorted row 502-3, and a fourth distorted row 502-4. As shown, the distorted rows 502 are distorted in a barrel distortion manner (e.g., due to lens distortion). Although barrel lens distortion is shown, the distortion of the distorted input image 500 may be any distortion caused by the shape of the lens or instability during the capture of the distorted input image 500. In addition, although gaps are shown between each of the distorted rows 502, the gaps should not be interpreted as representing anything other than arbitrary distortions of the input image (e.g., barrel distortion). Alternatively or additionally, for purposes of brevity, clarity, or explanation of COWE, gaps may not be present in the distorted input image or may be included in the distorted input image. Figure 5 and subsequent diagrams.

[0048] Figure 6 As generally shown at 600, COWE receives data from a system memory (e.g., Figure 1 and / or Figure 2 An example of loading pixel information into system memory 108). Figure 6 Including from Figure 5 The third warp row 502-3 and the fourth warp row 502-4 are overlapped on the grid 602 of the MAU. Figure 6 is shown as a rectangle with a solid white background or a shaded background. Figure 6 The first warp row 502-1 and the second warp row 502-2 are omitted. The MAU includes a storage device (eg, Figure 1 and / or Figure 2 602). The MAUs in the grid of MAUs 602 may be a number of pixels high and a number of pixels wide. A pixel may include a number of bits (e.g., eight bits) that may represent color information or gamma information for the pixel. For example, the grid of MAUs 602 may be viewed as an overlay of an image sensor of a camera that stores pixel information associated with pixels of an input image. Thus, the location of the MAUs within the grid 602 is not important, except that the location of the MAUs within the grid 602 is associated with pixel information for pixels in the same location of the input image.

[0049] COWE can distort the input image (e.g., Figure 3 The distorted input image 310, Figure 5 The distorted input image 500) is corrected to a corrected output image (eg, Figure 3 The corrected output image 316, Figure 4400 of the corrected output image). COWE may only acquire MAUs that include pixel information of the distorted input image. As shown, the MAUs not acquired by COWE are unacquired MAUs 604 and are represented by a pure white background. The unacquired MAUs may not contain pixel information associated with pixels of the distorted input image. For example, the lens of a camera may focus light onto an image sensor in a circular manner to capture pixel information of the distorted input image. The lens may focus light onto an area of ​​the image sensor that is not associated with the unacquired MAUs 604. Alternatively, the MAUs acquired by COWE are acquired MAUs 606 and are represented by a shaded background. Continuing with this example, the acquired MAUs 606 may contain pixel information associated with pixels of the distorted input image. That is, the lens of the camera may focus light onto an area of ​​the image sensor that is associated with the acquired MAUs 606. Therefore, the grid 602 of MAUs includes a plurality of unacquired MAUs 604 and a plurality of acquired MAUs 606. The pixel information included in the acquired MAU 606 may be used by COWE to interpolate pixel information for pixels of the corrected output image.

[0050] Figure 7 As generally shown at 700, COWE may be retrieved from a system memory (e.g., Figure 1 and / or Figure 2 Various examples of MAUs obtained from the system memory 108 (e.g., Figure 6 Unacquired MAU 604, Figure 6 606). Figure 7 , each of the MAUs is delimited by a number at the top of the MAU and a number to the left of the MAU. These numbers represent the pixel count that defines the shape (e.g., square, rectangle) and size (in pixels) of the MAU. The data size of the MAU in bits (b) or bytes (B) (e.g., eight bits) may depend on the specific pixel information of the pixels of the MAU and / or the format of the system memory. For example, the data may include color depth (e.g., 8-bit color, 16-bit color) and gamma (e.g., brightness) information, depending on the image (e.g., Figure 5 The distorted input image 500, Figure 3 The corrected output image 316, Figure 4 The total data size of the MAU is equal to the number of pixels in the MAU (eg, height times width, in pixels) times the data size of each pixel.

[0051] The first MAU 702 of various examples of MAUs is a square MAU that is 16 pixels wide and 16 bits high, for a total of 256 pixels. As an example, if the pixel information uses 8-bit color, the first MAU 702 includes 2,048 bits (e.g., 256 B) of data. The second MAU 704 is a rectangular MAU that is 32 pixels wide and 8 pixels high, for a total of 256 pixels. As an example, if 8 bits are used to describe the pixel information, the second MAU 704 includes 1,024 bits (e.g., 128 B) of data. The second MAU 704 that is 32 pixels wide and 8 pixels high can be used in frame rate control (FRC) applications. The third MAU 706 is 16 pixels wide and 8 pixels high, for a total of 128 pixels. The fourth MAU 708 is 16 pixels wide and 4 pixels high, for a total of 74 pixels. The fifth MAU 710 is a rectangular MAU that is 64 pixels wide and one pixel high. The sixth MAU 712 is half the size of the fifth MAU 710, being 32 pixels wide and one pixel high. The seventh MAU 714 is the same size as the sixth MAU 712, but is only 16 pixels wide and two pixels high. The eighth MAU 716 is half the size of the seventh MAU 714, being 16 pixels wide and only one pixel high.

[0052] Figure 8 At 800, a COWE (e.g., Figure 1 and / or Figure 2 COWE 114) is loaded into a cache memory (e.g., Figure 1 and / or Figure 2 The first cache memory 104 and / or the second cache memory 106) in the MAU (e.g., Figure 7 An example of any of MAUs 702 to 716). Figure 8 Including from Figure 5 The third warp row 502-3 and the fourth warp row 502-4 are overlapped onto the grid 802 of MAUs. As shown, there are three categories of MAUs, including inter-row MAUs 804, intra-row MAUs 806, and unobtained MAUs 808 (e.g., Figure 6 COWE can be performed by following the coordinate sequence (e.g., reference Figure 1 The coordinate sequence described by ) is used to obtain the MAU.

[0053] The coordinate sequence may include an intra-row coordinate sequence that progresses from bottom to top and from left to right or from right to left within a row, and an inter-row coordinate sequence that progresses from the top row to the bottom row in a row-by-row manner. In other words, the pixel information of the top row may be scanned following the intra-row coordinate sequence that progresses from bottom to top and from left to right or from right to left within the top row. Then, the same intra-row coordinate sequence may be used to scan the pixel information of the row adjacent to the top row and below the top row. The pixel information of subsequent rows may be scanned similarly. Alternatively, the coordinate sequence may include an intra-row coordinate sequence that progresses from bottom to top and from left to right or from right to left within a row, and an inter-row coordinate sequence that progresses from the bottom row to the top row in a row-by-row manner. It should be noted that because of distorted rows (e.g., Figure 5 The third warp row 502-3) is connected to the warped input image (eg, Figure 5 The distorted input image 500 is as wide as the image 500, so the sequence of row coordinates may proceed from top to bottom or bottom to top rather than from left to right or right to left. For clarity and brevity, Figure 8 A coordinate sequence 810 is shown, which includes an intra-row coordinate sequence from bottom to top and from left to right and an inter-row coordinate sequence from top to bottom.

[0054] COWE may scan pixels of the third warp row 502-3 and the fourth warp row 502-4 and retrieve the associated MAUs from the system memory. In this example, COWE starts from the left side of the third warp row 502-3 and scans along the scan line 810-1 (e.g., reference Figure 1 806-3.

[0055] COWE may use a cacheable read command to retrieve the first inter-row MAU 804-1, the first intra-row MAU 806-1, the second intra-row MAU 806-2, and the third intra-row MAU 806-3. For example, COWE may retrieve the first inter-row MAU 804-1 and load it into a computing device (e.g., Figure 1 and / or Figure 2 A second cache memory (e.g., Figure 1 and / or Figure 2 Similarly, COWE may obtain the first intra-row MAU 806-1, the second intra-row MAU 806-2, and the third intra-row MAU 806-3 and load them into the first cache memory of the computing device (e.g., Figure 1 and / or Figure 2 COWE may repeat this process, where the first MAU of the scan line (e.g., the bottom MAU, the first inter-row MAU 804-1, the second inter-row MAU 804-2) is loaded into the second cache memory, and the subsequent MAUs of the scan line (e.g., the intra-row MAU 806) are loaded into the first cache memory. By doing so, COWE may increase the speed of processing a distorted input image into a corrected output image as described in detail below.

[0056] As an example, the MAU of grid 802 may be eight pixels high by 32 pixels wide, for a total of 256 pixels. In addition, COWE may utilize bicubic interpolation on the distorted input pixels to determine the corrected output pixels. That is, COWE may bicubicly interpolate the distorted input pixels into the corrected output pixels based on the distorted input pixels and the neighboring distorted input pixels within a two-pixel radius. When COWE scans pixels along scan line 810, COWE may first search for the MAU containing pixel information in the first cache memory, and then, if the MAU does not exist in the first cache memory, search for the MAU containing pixel information in the second cache memory. In this example, the pixel information of the MAU associated with the first scan line 810-1 is loaded into the first cache memory and the second cache memory. Therefore, COWE may access the pixel information for bicubic interpolation from the first cache memory or the second cache memory, thereby significantly reducing the access time of the pixel information.

[0057] As shown, Figure 8 Also included are a first distorted input pixel 812-1 and a second distorted input pixel 812-2 (not to scale). As COWE scans pixels along the second scan line 810-2, COWE may load the MAU associated with the first distorted input pixel 816-1. As shown, the first distorted input pixel 812-1 is at the left edge of the intra-row MAU 806, meaning that pixel information for neighboring pixels within a two-pixel radius to the left is included in the first intra-row MAU 806-1 stored in the first cache memory. COWE may access pixel information from the first cache memory, which may be significantly faster (e.g., 100 times) than accessing the same pixel information from the system memory. Therefore, COWE may use the neighboring pixel information to bicubicly interpolate the first distorted input pixel 812-1 into the corrected output pixel more quickly. Because COWE may access pixel information of the intra-row MAU 806 more frequently than pixel information of the inter-row MAU 804, pixel information of the intra-row MAU 806 is stored in the first cache memory.

[0058] Although not shown, COWE may continue to scan pixels within the third warp row 502-3 using an intra-row coordinate sequence that proceeds from bottom to top and from left to right. COWE may scan the rightmost pixel information of the third warp row 502-3, and then proceed to scan (e.g., using the same intra-row coordinate sequence) pixel information within the fourth warp row 502-4. In this manner, COWE may use an inter-row coordinate sequence that proceeds from top to bottom. In other words, COWE may scan pixel information within an upper row (e.g., the third warp row 502-3) using an intra-row coordinate sequence, and then scan pixel information within an adjacent row below the upper row (e.g., the fourth warp row 502-4) using an intra-row coordinate sequence.

[0059] As shown, the second distorted input pixel 812-2 is the topmost pixel of the intra-row MAU 806 within the fourth distorted row 502-4. Therefore, in order to process the second distorted input pixel 812-2 into a corrected output pixel, COWE must perform bicubic interpolation of pixel information of neighboring pixels within a two-pixel radius. In this example, COWE may access neighboring pixel information above the second distorted input pixel 812-2 within the second cache memory. This is because the first inter-row MAU 804-1 was previously loaded into the second cache memory when COWE scanned the pixel information along the first scan line 810-1. Therefore, COWE may process the second distorted input pixel 812-2 into a corrected output pixel faster (e.g., 10 times) than if COWE accessed the same neighboring pixel information from the system memory. Although not shown, COWE may similarly process another distorted input pixel into another corrected output pixel using pixel information of the second inter-row MAU 804-2, which may be stored in the second cache memory.

[0060] Furthermore, due to the coordinate sequence, COWE can optimize the first cache memory and the second cache memory. The optimization of the first cache memory can be described by Equation 1.

[0061] In equation 1, is the size of the first cache memory, is the height of the corrected output line, is a MAU shape, where is the short side (e.g., height), and is the long side (e.g., width), and is the bicubic interpolation window size. Similarly, the optimization of the second cache memory can be described by Equation 2.

[0062] In equation 2, is the size of the first cache memory, is the bottom line of the pixel (e.g., the bottom pixel within the inter-row MAU 812), For interpolation kernel expansion, is a MAU shape, where is the height (e.g., eight pixels) and w is the width (e.g., 32 pixels), and is the corrected output image shape, where is the height (e.g., 1,080 pixels) and is the width (e.g., 1,920 pixels).

[0063] As an example, the MAU shape may be eight pixels high by 32 pixels wide, the corrected output image may be 1,080 pixels high by 1,920 pixels wide, each row dividing the corrected output image may be 108 pixels high, and each pixel (e.g., color and gamma) may be described by eight bits. Continuing with this example, the size of the first cache memory may be optimized to 7,680 pixels or 7,680 B (e.g., 7,680 pixels x 8 bits per pixel). Similarly, the size of the second cache memory may be optimized to 99,840 pixels or 99,840 B (e.g., 99,840 pixels x 8 bits per pixel). Example Method

[0064] Fig. 9 Describes how COWE can distort input images (e.g. Figure 3 Example method 900 performed by distorting the input image 310). At 902, having COWE, a first cache memory (e.g., Figure 1 and / or Figure 2 The first cache memory 104 and the second cache memory (eg, Figure 1 and / or Figure 2 The computing device of claim 1 receives a distorted input image including a plurality of pixel information from a second cache memory 106 of the first cache memory. The first cache memory may be an L1 cache, which is about 100 times faster than a system memory (e.g., DRAM, double data rate (DDR) DRAM). The second cache memory may be an L2 cache, which is about 10 times faster than the system memory. During capture of the distorted input image, the distorted input image may be distorted due to lens distortion (e.g., of a camera) or instability (e.g., instability, long exposure time).

[0065] At 904, COWE divides the warped input image including the plurality of pixel information into N rows. N may be a positive integer including four, eight, 10, 12, 16, 40, etc. Depending on the computing power of the computing device and / or the format of the system memory of the computing device, the plurality of pixel information may be included in a MAU of a certain size (in pixels).

[0066] At 906, COWE scans a plurality of pixel information of the distorted input image following a coordinate sequence comprising N rows. COWE may scan pixels by accessing MAUs associated with pixels from a system memory of a computing device. COWE may, for example, access individual MAUs or multiple MAUs from a system memory. The coordinate sequence may include an intra-row coordinate sequence that is at least one of from bottom to top and from left to right or from bottom to top and from right to left, and an inter-row coordinate sequence that is from top to bottom. Alternatively, the coordinate sequence may include an intra-row coordinate sequence that is at least one of from top to bottom and from left to right or from top to bottom and from right to left, and an inter-row coordinate sequence that is from bottom to top.

[0067] At 908, COWE loads a first portion of the plurality of pixel information of the warped input image into a first cache memory of the computing device. The first portion may include one or more in-line MAUs accessed by COWE from a system memory of the computing device.

[0068] At 910, COWE determines a first corrected portion of multiple pixel information of a corrected output image based on a first portion of multiple pixel information of the distorted input image stored in a first cache memory. COWE may determine the first corrected portion by determining a position of the corrected portion of the multiple pixel information in the corrected output image using a transformation model. The transformation model may be a predetermined transformation function based on, for example, the curvature of a lens of a camera that captured the distorted input image. Alternatively or additionally, the transformation model may be based on sensor data (e.g., accelerometer data) captured simultaneously with the distorted input image. The sensor data may be used by COWE to determine the first corrected portion via an EIS process.

[0069] At 912, COWE loads a second portion of the plurality of pixel information of the warped input image into a second cache memory of the computing device. The second portion may include one or more inter-row MAUs accessed by COWE from a system memory of the computing device.

[0070] At 914, COWE determines a second corrected portion of the plurality of pixel information of the corrected output image based on a second portion of the plurality of pixel information of the warped input image stored in the second cache memory. COWE may determine the second corrected portion using a transformation model and / or sensor data as described above.

[0071] At 916, COWE stores the first and second portions of the plurality of pixel information of the corrected output image as the output image in a system memory of the computing device. The first and second portions may include subsequent first and second portions of the corrected output image determined by COWE by processing all portions of the warped input image following the coordinate sequence. The system memory may be a DDR DRAM or an SSD of the computing device.

[0072] By executing the above example method 900, COWE can use the first cache memory and the second cache memory to correct the distorted input image into a corrected output image faster than when COWE uses only the system memory. In addition, COWE can optimize the size of the first cache memory and the second cache memory by following the above coordinate sequence. By doing so, COWE improves the user experience of the computing device through the smaller physical size and higher power efficiency of the first cache memory and the second cache memory. Additional Examples

[0073] In the following sections, additional examples are provided.

[0074] Example 1: A method, the method comprising: receiving a warped input image comprising a plurality of pixel information at a computing device having a cache-optimized warp engine, a first cache memory, and a second cache memory; dividing the warped input image comprising the plurality of pixel information into N rows by the cache-optimized warp engine; scanning the plurality of pixel information of the warped input image by the cache-optimized warp engine following a coordinate sequence comprising the N rows, the coordinate sequence comprising at least one of: an intra-row coordinate sequence, the intra-row coordinate sequence being at least one of the following: from bottom to top and from left to right; or from bottom to top and from right to left; and an inter-row coordinate sequence, the inter-row coordinate sequence being from top to bottom; or an intra-row coordinate sequence, the intra-row coordinate sequence being at least one of the following: from top to bottom and from left to right; or from top to bottom and from right to left; and an inter-row coordinate sequence, the inter-row coordinate sequence being from bottom to top; dividing the plurality of pixel information of the warped input image into N rows by the cache-optimized warp engine; scanning the plurality of pixel information of the warped input image by the cache-optimized warp engine following a coordinate sequence comprising the N rows A first portion of pixel information is loaded into the first cache memory of the computing device; a first corrected portion of multiple pixel information of a corrected output image is determined by the cache-optimized warp engine based on the first portion of the multiple pixel information of the warped input image stored in the first cache memory; a second portion of the multiple pixel information of the warped input image is loaded into the second cache memory of the computing device by the cache-optimized warp engine; a second corrected portion of the multiple pixel information of the corrected output image is determined by the cache-optimized warp engine based on the second portion of the multiple pixel information of the warped input image stored in the second cache memory; and the first portion and the second portion of the multiple pixel information of the corrected output image are stored as the output image in a system memory of the computing device or in a system memory associated with the computing device by the cache-optimized warp engine.

[0075] Example 2: The method of Example 1, wherein: the distorted input image is W pixels wide and T pixels high; W is an integer greater than one; and T is an integer greater than one.

[0076] Example 3: A method as described in Example 2, wherein: each of the N rows is W pixels wide; and each of the N rows is an integer division of T pixels high.

[0077] Example 4: The method as in any of the preceding examples, wherein the coordinate sequence comprises an intra-row coordinate sequence from bottom to top and from left to right and an inter-row coordinate sequence from top to bottom.

[0078] Example 5: A method as described in any of the preceding examples, wherein: the first cache is an L1 cache of the cache-optimized warp engine; and the second cache is an L2 cache of the cache-optimized warp engine; or the second cache is a system-level cache of a system-on-chip of the computing device.

[0079] Example 6: A method as described in any of the preceding examples, wherein at least one of the following: determining the first portion of the multiple pixel information of the corrected output image using an arbitrary transformation function; or determining the second portion of the multiple pixel information of the corrected output image using the arbitrary transformation function.

[0080] Example 7: A method as described in any of the preceding examples, wherein determining the first portion or the second portion of the plurality of pixel information of the corrected output image uses at least one of: piecewise interpolation; linear interpolation; polynomial interpolation; spline interpolation; or simulated interpolation.

[0081] Example 8: A method as described in any of the preceding examples, wherein determining the first portion or the second portion of the multiple pixel information of the corrected output image uses at least one of: lens distortion; geographic distortion; motion compensation; electronic image stabilization; rolling shutter correction; or camera calibration.

[0082] Example 9: A method as described in any of the preceding examples, wherein at least one of the following: scanning the multiple pixel information of the warped input image includes scanning multiple memory access units; loading the first part of the multiple pixel information of the warped input image into the first cache memory of the computing device includes loading at least one memory access unit of the multiple memory access units; or loading the second part of the multiple pixel information of the warped input image into the second cache memory of the computing device includes loading at least one memory access unit of the multiple memory access units.

[0083] Example 10: The method of Example 9, wherein: the at least one memory access unit is L pixels long and S pixels wide; L is an integer greater than one; and S is an integer greater than one.

[0084] Example 11: The method of Example 10, wherein S is less than L.

[0085] Example 12: A method as described in any of the preceding examples, wherein: the pixels of the distorted input image include M bits; and M is at least one of the following: an integer greater than or equal to one; four bits; eight bits; 16 bits; 32 bits; or 64 bits.

[0086] Example 13: A method as described in any of the preceding examples, wherein at least one of the following: the multiple pixel information of the distorted input image includes: position information; gamma information; or color information; or the multiple pixel information of the corrected output image includes: position information; gamma information; or color information.

[0087] Example 14: A computing device comprising: a first cache memory; a second cache memory; a system memory; at least one processor; and a computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to implement a cache-optimized warp engine by utilizing the first cache, the second cache, and the system memory by executing the method described in any one of Examples 1 to 13.

[0088] Example 15: Computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of Examples 1 to 13. in conclusion

[0089] Unless the context dictates otherwise, the use of the word "or" herein may be considered use of an "inclusive or" or a term that permits inclusion or application of one or more items connected by the word "or" (e.g., the phrase "A or B" may be interpreted as permitting only "A," as permitting only "B," or as permitting both "A" and "B"). In addition, as used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. For example, "at least one of a, b, or c" may encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c). In addition, the items represented in the drawings and the terms discussed herein may indicate one or more items or terms, and thus may refer interchangeably to the singular or plural forms of the items and terms in this written description.

[0090] Although implementations of cache optimized warp engine systems and techniques and devices for implementing cache optimized warp engine have been described in language specific to certain features and / or methods, the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of decoding RF signal reflections into object embedding for context triggering.

Claims

1. A method comprising: At a computing device having a cache-optimized warp engine, a first cache memory, and a second cache memory, receiving a warped input image including a plurality of pixel information; Dividing, by the cache-optimized warping engine, the warped input image containing the plurality of pixel information into N rows; The cache-optimized warping engine scans the plurality of pixel information of the warped input image according to a coordinate sequence including the N rows, wherein the coordinate sequence includes at least one of the following: In-line coordinate sequence, the in-line coordinate sequence is at least one of the following: from bottom to top and from left to right; or from bottom to top and from right to left; and A sequence of inter-row coordinates, the sequence of inter-row coordinates is from top to bottom; or In-line coordinate sequence, the in-line coordinate sequence is at least one of the following: from top to bottom and from left to right; or from top to bottom and from right to left; and A row coordinate sequence, wherein the row coordinate sequence is from bottom to top; Loading, by the cache-optimized warping engine, a first portion of the plurality of pixel information of the warped input image into the first cache memory of the computing device; determining, by the cache optimized warping engine, a first corrected portion of a plurality of pixel information of a corrected output image based on the first portion of the plurality of pixel information of the warped input image stored in the first cache memory; Loading, by the cache-optimized warping engine, a second portion of the plurality of pixel information of the warped input image into the second cache memory of the computing device; determining, by the cache optimized warping engine, a second corrected portion of the plurality of pixel information of the corrected output image based on the second portion of the plurality of pixel information of the warped input image stored in the second cache memory; as well as The first portion and the second portion of the plurality of pixel information of the corrected output image are stored by the cache optimized warping engine as the output image in a system memory of the computing device or in a system memory associated with the computing device.

2. The method of claim 1, wherein: The distorted input image is W pixels wide and T pixels high; W is an integer greater than one; and T is an integer greater than one.

3. The method of claim 2, wherein: Each of the N rows is W pixels wide; and Each of the N rows is an integer division of T pixels high. 4 . The method according to claim 1 , wherein the coordinate sequence comprises an intra-row coordinate sequence from bottom to top and from left to right and an inter-row coordinate sequence from top to bottom.

5. A method as claimed in any one of the preceding claims, wherein: The first cache is an L1 cache of the cache-optimized warp engine; and The second cache is an L2 cache of the cache-optimized warp engine; or The second cache is a system-level cache of a system on chip of the computing device.

6. The method of any one of the preceding claims, wherein at least one of the following: determining the first portion of the plurality of pixel information of the corrected output image using an arbitrary transformation function; or Determining the second portion of the plurality of pixel information of the corrected output image uses the arbitrary transformation function.

7. The method of any one of the preceding claims, wherein determining the first portion or the second portion of the plurality of pixel information of the corrected output image uses at least one of: Segmented interpolation; Linear interpolation; Polynomial interpolation; Spline interpolation; or Analog interpolation.

8. The method of any one of the preceding claims, wherein determining the first portion or the second portion of the plurality of pixel information of the corrected output image uses at least one of: Lens distortion; Geographic distortion; Motion compensation; Electronic image stabilization; Rolling shutter correction; or Camera calibration.

9. The method of any one of the preceding claims, wherein at least one of the following: Scanning the plurality of pixel information of the warped input image includes scanning a plurality of memory access units; Loading the first portion of the plurality of pixel information of the warped input image into the first cache memory of the computing device comprises loading at least one memory access unit of the plurality of memory access units; or Loading the second portion of the plurality of pixel information of the warped input image into the second cache memory of the computing device includes loading at least one memory access unit of the plurality of memory access units.

10. The method of claim 9, wherein: The at least one memory access unit is L pixels long and S pixels wide; L is an integer greater than one; and S is an integer greater than one. The method of claim 10 , wherein S is less than L.

12. A method as claimed in any one of the preceding claims, wherein: A pixel of the warped input image comprises M bits; and M is at least one of the following: an integer greater than or equal to one; Four bits; Eight bits; 16 bits; 32 bits; or 64 bits.

13. The method of any one of the preceding claims, wherein at least one of the following: The plurality of pixel information of the distorted input image comprises: Location information; Gamma information; or Color information; or The plurality of pixel information of the corrected output image includes: Location information; Gamma information; or Color information.

14. A computing device, the computing device comprising: a first cache memory; a second cache memory; System memory; at least one processor; and A computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to implement a cache-optimized warp engine by utilizing the first cache, the second cache, and the system memory by performing the method of any one of claims 1 to 13.

15. A computer readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 13.

Images