Systems and methods for efficient image transformation operations

Abstract

Methods, systems, and apparatus for receiving a request to perform a plurality of image transformation operations on an input image to generate a plurality of output image frames. An identifier is assigned to each respective image transformation operation and a corresponding output image frame. Image data for the input image is obtained from memory and stored in a local cache. An output image frame portion is generated based on a portion of image data for the input image stored in the local cache and an image transformation operation corresponding to the identifier of the respective output image frame. The output image frame portion is stored in memory within a separate container associated with each of the plurality of output image frames.

Description

[0001] Atorney Docket No.: 56113-0781WO1

[0002] SYSTEMS AND METHODS FOR EFFICIENT IMAGE TRANSFORMATION

[0003] OPERATIONS

[0004] TECHNICAL FIELD

[0005] This specification relates to image processing, and more particularly to systems and methods for efficient image transformation operations using a single input image to generate multiple output images.

[0006] BACKGROUND

[0007] Image processing operations can include generation of multiple output images based on a set of input images. Examples of such image processing operations can include two- dimensional (2D) to three-dimensional (3D) conversion, frame rate increase, depth-based blurring, etc. In 2D to 3D conversion operations, for example, an input image frame is processed to generate several output image frames at various depths. In frame rate increase operations, one or more new image frames are inserted between image frames currently present in the media. In one approach, the one or more new image frames can be generated based on interpolating pixels values from one or more already present image frames.

[0008] Hardware resources utilized for carrying out the image processing operations can include digital signal processors, graphics processing units, image signal processors, etc. Traditional approaches to utilizing hardware resources to cany7out image processing operations include duplicating hardware resources to generate the multiple output images or utilizing the hardware resources to serially generate the multiple output images. For example, in one traditional approach, the hardware resources are duplicated to generate multiple output images in parallel from the set of input images. That is, each duplicated hardware resource can apply a respective image transformation operation to the set of input images to generate a respective output image frame. While this approach can generate the output image frames with high throughput, additional silicon area is needed to accommodate the duplicated hardware resources. In addition, each of the duplicated hardware resources will have its own separate memory read operation to read its respective portion of the input images. The multiple hardware resources along with the multiple memory read operations to obtain the same input image data can result in high power consumption. In the other traditional approach, hardware resources are utilized to sequentially generate the output images one after another. In generating the multiple output images sequentially, memory7is Atorney Docket No.: 56113-0781WO1 also accessed sequentially to read the entire input image data. That is, input image data is read from memory each time an output image is generated. This is not only inefficient but also results in high power consumption due to repeated memory read operations.

[0009] SUMMARY

[0010] This specification describes techniques for transforming input image frames into output image frames. In particular, the techniques relate to image processing operations that utilize image transformation operations to generate output image frames from a set of input image frames. An image transformation engine can be utilized to generate the output image frames based on the image transformation operations. The image transformation engine can include an output generation unit coupled with a local cache. The output generation unit generates output image frame portions based on image data of a portion of an input image and an image transformation operation. These output image frame portions correspond to a plurality of output image frames that are to be generated. The output generation unit is configured such that output image frame portions at similar locations in their respective output image frames are generated in sequence. That is, the output generation unit generates an output image frame portion at a first location in a first output image frame, followed by generating an output image frame portion at the first location in a second output image frame, and so on until output image frames at the same location in all output image frames are generated. In some examples, generating output image frame portions at similar locations in the output image frames may utilize common input image data. This common input image data can be stored in the local cache coupled with the output generation unit. As a result, the probability of finding the desired input image data in the local cache when the output generation unit generates subsequent output image frame portions is increased. Consequently, the number of times the image transformation engine accesses main memory to read input image data is reduced, thereby increasing the performance of the image transformation operation.

[0011] The image transformation engine can utilize an axis mapper to generate coordinates of input pixels in the input image based on locations of the output image frame portions being generated. The axis mapper can then generate memory locations associated with those coordinates to access the input image data. In some instances, the coordinates generated for the input pixels may not match with the actual positions of the input pixels in the input image. In such instances, the output generation unit can generate the output image frame portion by Atorney Docket No.: 56113-0781WO1 approximating the value of the output image frame portion based on values of neighboring input pixels.

[0012] Particular examples of the subject mater described in this specification can be implemented so as to realize one or more of the following advantages. As noted above, some traditional approaches to generating the two or more image frames from a set of input image frames can duplicate the computing resources needed to generate one image frame to speedup the generation of the multiple image frames (i.e., use multiple output image generation engines to generate the multiple output images concurrently). While this approach can increase performance, the increased performance comes at a cost of increased power consumption due to memory read operations carried out by each duplicated computing resource and increased silicon area (to accommodate each of the duplicated computing resources). Some other approaches can use multiplexing techniques to generate multiple output image frames from a set of input image frames by reusing the same computing resources. Such approaches, however, have high latency (due to repeated memory read operations to load input image data) and low throughput and also suffer from high power consumption due to multiple read operations from memory. The techniques discussed herein utilize fewer computing resources while providing high throughput and low memory read power consumption. In particular, the techniques discussed herein employ a single output generation unit as implemented within an image transformation engine. The output image frames are generated by processing similarly located portions of the output image frames in a sequence. This approach to generating the output image frames increases the probability7that the input image data needed to generate the output image frames is present in a local cache coupled with the output generation unit. As a result, memory read operations are reduced at least because the relevant input image data to be processed in a particular iteration of generating an output image frame portion is likely to be present in the local cache, thereby reducing the power that would otherwise be consumed in repeatedly accessing input mage data from memory. In addition, as the high latency memory7operations are reduced, the performance of the image transformation engine is improved.

[0013] The techniques discussed herein improve performance compared to both traditional approaches discussed above. Specifically, compared to the traditional approach in which computing resources are duplicated, the techniques discussed herein, by using a single computing resource (the image transformation engine) without duplication, occupy relatively less silicon area and consume relatively less power by reducing the computing resources and by reducing the number of memory read operations. Further, compared to the traditional Atorney Docket No.: 56113-0781WO1 approach in which a single computing resource is used to serially generate the output image frames, the techniques discussed herein, albeit occupying similar silicon area, reduce latency and reduce power consumption by reducing the number of memory read operations.

[0014] In one aspect, the disclosure relates to a method. The method includes receiving a request to perform a plurality’ of image transformation operations on an input image to generate a plurality- of output image frames; assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated. The method also includes storing image data for the input image, obtained from memory, in a local cache. The method also includes generating, using a portion of the image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality of output image frames to be generated, where each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame. The method also includes storing, in memory and within a separate container associated with each of the plurality of output image frames, the respective output image frame portions.

[0015] Implementations may include one or more of the following features. The method where the respective output image frame portion includes an output pixel, the method may include: determining location of one or more input pixels in the input image based on a location of the output pixel in the respective output image frame and the respective image transformation operation; reading from the local cache the one or more input pixels based on the location of the one or more input pixels; and generating the output pixel based on the one or more input pixels. The method may include: generating an output pixel data structure including a value of the output pixel and the respective identifier corresponding to the respective output image frame on which the output pixel is located; and storing the value of the output pixel within the separate container associated yvith the respective output image frame. The image transformation operation includes a displacement value, and a location of the portion of the image data on the input image is determined based on a location of output image frame portion on the respective output image frame and the displacement value. The location of the portion of the image data on the input image includes a pl urality of input pixels on the input image, the method may include determining a value for an output pixel in the respective output image frame portion based on interpolating values of the plurality of input pixels. The method may include generating output image frame portions of the plurality of output image frames in successive single clock cycles. At least one image transformation Atorney Docket No.: 56113-0781WO1 operation of the plurality of image transformation operations includes a displacement value determined based on displacement values at a set of grid points located at positions adjacent to a position of the respective output image frame portion. The method may include: generating the respective output image frame portions located at a same position in their respective plurality of output image frames in a first sequence, and generating another set of respective output image frame portions located at another same position in the respective plurality of output image frames in a second sequence subsequent to the first sequence.

[0016] One general aspect includes one or more non-transitoiy computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to perform operations for determining buffering components for placement between two nodes on an integrated circuit design. The operations include receiving a request to perform a plurality of image transformation operations on an input image to generate a plurality of output image frames; assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated. The operations also include storing image data for the input image, obtained from memory, in a local cache. The operations also include generating, using a portion of the image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality of output image frames to be generated, where each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame. The operations also include storing, in memory and within a separate container associated with each of the plurality of output image frames, the respective output image frame portions.

[0017] Implementations may include one or more of the following features. The computer- readable storage media where the respective output image frame portion includes an output pixel, the operations may include: determining location of one or more input pixels in the input image based on a location of the output pixel in the respective output image frame and the respective image transformation operation; reading from the local cache the one or more input pixels based on the location of the one or more input pixels; and generating the output pixel based on the one or more input pixels. The computer-readable storage media the operations may include generating an output pixel data structure including a value of the output pixel and the respective identifier corresponding to the respective output image frame on which the output pixel is located; and storing the value of the output pixel within the separate container associated with the respective output image frame. The image transformation operation includes a displacement value, and a location of the portion of the Atorney Docket No.: 56113-0781WO1 image data on the input image is determined based on a location of output image frame portion on the respective output image frame and the displacement value. The location of the portion of the image data on the input image includes a plurality of input pixels on the input image, the operations may include determining a value for an output pixel in the respective output image frame portion based on interpolating values of the plurality of input pixels. The computer-readable storage media the operations may include generating output image frame portions of the plurality of output image frames in successive single clock cycles. At least one image transformation operation of the plurality of image transformation operations includes a displacement value determined based on displacement values at a set of grid points located at positions adjacent to a position of the respective output image frame portion. The computer- readable storage media the operations may include generating the respective output image frame portions located at a same position in their respective plurality of output image frames in a first sequence, and generating another set of respective output image frame portions located at another same position in the respective plurality' of output image frames in a second sequence subsequent to the first sequence.

[0018] One general aspect includes a system. The system also includes one or more processors. The system also includes one or more storage devices storing instructions that when executed by the one or more processors to perform operations for determining buffering components for placement between two nodes on an integrated circuit design, the operations may include receiving a request to perform a plurality of image transformation operations on an input image to generate a plurality of output image frames. The system also includes assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated. The operations also include storing image data for the input image, obtained from memory, in a local cache. The operations also include generating, using a portion of the image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality of output image frames to be generated, where each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame. The operations also include storing, in memory and within a separate container associated with each of the plurality of output image frames, the respective output image frame portions.

[0019] Implementations may include one or more of the following features. The sy stem where the respective output image frame portion includes an output pixel, the operations may include: determining location of one or more input pixels in the input image based on a Atorney Docket No.: 56113-0781WO1 location of the output pixel in the respective output image frame and the respective image transformation operation; reading from the local cache the one or more input pixels based on the location of the one or more input pixels; and generating the output pixel based on the one or more input pixels. The operations may include generating an output pixel data structure including a value of the output pixel and the respective identifier corresponding to the respective output image frame on which the output pixel is located; and storing the value of the output pixel within the separate container associated with the respective output image frame. The image transformation operation includes a displacement value, and a location of the portion of the image data on the input image is determined based on a location of output image frame portion on the respective output image frame and the displacement value. The location of the portion of the image data on the input image includes a plurality of input pixels on the input image, the operations may include determining a value for an output pixel in the respective output image frame portion based on interpolating values of the plurality of input pixels. The system the operations may include generating output image frame portions of the plurality of output image frames in successive single clock cycles. At least one image transformation operation of the plurality of image transformation operations includes a displacement value determined based on displacement values at a set of grid points located at positions adjacent to a position of the respective output image frame portion. The operations may include generating the respective output image frame portions located at a same position in their respective plurality of output image frames in a first sequence, and generating another set of respective output image frame portions located at another same position in the respective plurality' of output image frames in a second sequence subsequent to the first sequence.

[0020] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0021] The details of one or more embodiments of the subject mater of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject mater will become apparent from the description, the drawings, and the claims. Atorney Docket No.: 56113-0781WO1

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a block diagram of a portion of an example integrated circuit design.

[0024] Figure 2 shows a functional block diagram of an image transformation engine.

[0025] Figures 3A and 3B depict example image transformation operations that can be carried out by the image transformation engine.

[0026] Figure 4 shows an example sequence of output image frame generation by the image transformation engine.

[0027] Figure 5 depicts a block diagram of an example image transformation engine.

[0028] Figure 6 shows a flow diagram of a process for generating a plurality of output image frames from a set of input image frames.

[0029] Figure 7 shows an example approach to generating the output image frame portion based on input pixel values.

[0030] Figure 8 shows an example grid-based transformation operation approach to determining displacement values for an output image frame portion.

[0031] Like reference numbers and designations in the various drawings indicate like elements.

[0032] DETAILED DESCRIPTION

[0033] FIG. 1 shows an example electronic device 100 that includes a system-on-chip 140 C’SoC 140”) and a memory device 170. The SoC 140 includes a central processing unit 104 (‘’CPU 104”), a memory controller 105, a memory 106, a resource manager 108, and an IP / circuit block 110. In other examples, the SoC 140 can include fewer or more components, e.g., the SoC 140 can include no CPU or IP / circuit block. In some implementations, the electronic device 100 can include multiple SoCs 140 and descriptions about the SoC 140 can apply equally to each of the multiple SoCs and / or processors that may be included at electronic device 100.

[0034] The CPU 104 can be a general -purpose CPU (e.g., a single or multi-core CPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory7and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of Atorney Docket No.: 56113-0781WO1 the application, or both. The memory 106 is a system memory, shared memory', or both. In the example of FIG. 1. memory 106 is depicted external to circuit block 110. However, memory 106 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory7of the SoC 140, such as static random-access memory (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0035] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block 110. The memory' 106 can also include various other ty pes of memory, such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0036] The resource manager 108 is implemented in hardware and software. Aspects of the resource manager 108 can be also implemented as firmware or firmware of a device of the SoC 140, such as a ML processor, the memory7controller 105, or the CPU 104. The resource manager 108 includes control logic implemented in hardware, software, or both. For example, the resource manager 108 can include resources such as flip-flops, registers, buffers, etc. that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software. In some implementations, the resource manager 108 is a decoding / encoding resource manager that generates control signals for executing decompression and compression operations of system 100.

[0037] The circuit block 1 10 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include an image signal processor (ISP) 112, a host processing unit (HPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary7hardware elements. For example, each of the ISP 112, HPU 114, DSP 116, and GPU 118 can be a respective proprietary7IP block (or IP device) of a particular entity or device manufacturer.

[0038] The HPU 1 14 can be a special-purpose processor, such as ML hardware accelerator, neural processor unit, neural network (or ML) processor, or another application-specific processor. In some implementations, the HPU 114 is a neural network tensor processor (e.g., a tensor processing unit (TPU)) with an integrated circuit architecture that is optimized for power-efficient execution of ML computations using tensor constructs such as multidimensional matrices / tensors. Atorney Docket No.: 56113-0781WO1

[0039] The circuit block 110 also can include an image transformation engine 120 and an image processing engine 126. The image transformation engine 120 performs transformations of input images into output images and is discussed in further detail in this application. The image transformation engine 120 can generate multiple output images based on multiple transformations of a common set of input image. Examples of such operations can include image frame interpolation, two-dimension (2-D) to three-dimension (3-D) conversion, depth-based blur effect, etc. The image processing engine 126 can carry out additional image processing tasks not carried out by the image transformation engine 120. The additional image processing tasks can include, for example, color adjustment, image upscaling / downscaling, filtering, etc.

[0040] In some examples, the image transformation engine 120 can be part of the image processing engine 126. That is, the image processing engine 126 can perform the functions and include the components of the image transformation engine 120 discussed herein. In some examples, the output images generated by the image transformation engine 120 can be provided to the image processing engine 126 for further processing. In some examples, similar to the image processing engine 126, the image transformation engine 120 also can communicate with one or more processors the ISP 112, the HPU 114, the DSP 116, the GPU 118 as well as other components of the SoC 140.

[0041] One or more aspects of the resource manager 108 can be implemented as a software routine (or module) of the CPU 104. which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from memory device 170 and / or from a system memory of the SoC 102, such as memory 106. In some implementations, each processor, e.g.. the ISP 112, the DSP 116, the HPU 114, the GPU 118, of the SoC 140 includes multiple cores.

[0042] The CPU 104 and / or the resource manager 108 can generate control signals to control and / or manage the operations of one or more other components of the electronic device 100, such as the image processing engine 126 and the image transformation engine 120. The control signals are routed at the electronic device 100 using an example bus 122 (“SoC bus 122”) of the SoC 140. The control signals can include commands, requests, data, instructions, or combination of these.

[0043] In some implementations, the resource manager 108 cooperates with the CPU 104 and memory controller 105 to dynamically control and manage one or more process-in-memory (PiM) operations, compute-in-memory (CiM) operations, or both. The operations involve Atorney Docket No.: 56113-0781WO1 accessing and processing data stored locally at the memory device 170. The PiM and CiM operations are executed locally at the memory device 170 based on control signaling generated at the SoC 140. The control signals are received at the memory device 170 and processed locally to trigger or initiate PiM / CiM operations using PiM / CiM compute elements of the memory device 170. The operations can be executed in support of heterogeneous computations that involve two or more processing units of the IP block 110, the CPU 104, or both. In some examples, the memory device 170 can include an example image processing engine (such as for example, the image processing engine 126 and / or the image transformation engine 120) to perform PiM / CiM operations at the memory device 170. That is, the image processing engine can be used to process images or other media that is stored at the memory device 170 or received at the memory device 170 via the memory controller 105, the disk memory 160, or a combination of these. In some cases, data can be encrypted or otherwise packaged for transmission.

[0044] The SoC 140 cooperates with the memory device 170 to perform computations using data (e.g., encoded data) obtained from memory banks of the memory device 170. The computations can be for operations or workloads that involve one or more of the processors at IP block 110. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In some implementations, the heterogeneous operation includes a host device, e g., HPU 114. and the memory device 170 cooperating to execute an example ML workload. In at least one example, the memory device 170 may be external to the SoC 140, whereas in another example the memory' device 170 may be internal to the SoC 140.

[0045] In the example of FIG. 1, the SoC 140 is an integrated circuit of an example user / client device 130, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 130a, tablet 130b, laptop 130c, smartwatch or wearable device 130d. The device 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the SoC 140 and memory device 170 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0046] The image processing engine 126 and the image transformation engine 120 shown as being collocated with processing resources of the IP block 110. However, in other examples, the image processing engine 126 and the image transformation engine 120 can be disposed at other locations of the SoC 140. The image processing engine 126 and the image transformation engine 120 can be implemented as a set of integrated circuits, or discrete Atorney Docket No.: 56113-0781WO1 hardware modules, connected to an SoC processor 140 and memory' device 170. For example, the image processing engine 126 and / or the image transformation engine 120 can be coupled to the memory device 170 through one or more memory channels, communication buses, or circuit interconnects in electronic device 100 or client device 130.

[0047] As discussed in more detail below, the image transformation engine 120 can be configured to efficiently carry out image transformation operations where multiple output image frames are generated based on a set of input images. In particular, the image transformation engine 120 is configured to sequentially generate similar portions of the multiple output image frames, which increases the probability that input image data utilized to generate a portion of one output image frame can be utilized to generate a portion of other output image frames, thereby reducing the number of memory operations needed to read input image data.

[0048] Figure 2 shows a functional block diagram of an image transformation engine 120. The image transformation engine 120 operates on a set of input images 202 based on a plurality of image transformation operations 204 and generates a plurality’ of output images 206. The set of input images 202 can include one or more images. In some examples, the set of input images 202 can include one or more image frames of a video. In some other examples, the set of input images 202 can include a single image frame corresponding to a still image. The plurality’ of image transformation operations 204 correspond to the generation of a plurality of output images 206. For example, the plurality' of image transformation operations 204 can include n image transformation operations (Op-1, Op- 2, . . . , Op- / ?). which the image transformation engine 204 can utilize to generate n output images (OIm-1, OIm-2, . . . . Olm-n).

[0049] Figures 3A and 3B depict example image transformation operations that can be carried out by the image transformation engine 120. Figure 3 A depicts a frame interpolation operation yvhile Figure 3B depicts a depth volume operation. Frame interpolation can be carried out in instances where the frame rate of the video is to be increased. For example, a video may have a frame rate of 30 frame per second, which is to be increased to 120 frames per second. The increase in frame rate can be accommodated by adding additional image frames to the existing image frames of the video. One approach to adding additional image frames is to interpolate the additional image frames between existing image frames of the video. Referring to Figure 3A, two existing image frames: a first existing image frame 302 and a second existing image frame 304, of the video are shown. The image transformation engine 120 can generate additional image frames 306 between the first existing image frame Atorney Docket No.: 56113-0781WO1

[0050] 302 and the second existing image frame 304. The image transformation engine 120 can generate the additional image frames 306 based on the image data of the first existing image frame 302, or the second existing image frame 304, or both the first and second existing image frames 302, 304. Each additional image frame can have an associated image transformation operation, which the image transformation engine 120 can utilize to transform the first and / or second existing image 302, 304, into the additional image frame. Generally, each additional image frame can have a unique image transformation operation.

[0051] Figure 3B depicts a 2D to 3D conversion example in which a still image 308 is converted into a 3D image by generating additional image frames 310 at various depth levels. The image transformation engine 120 can utilize a plurality of image transformation operations corresponding to the additional image frames 310 to transform the still image 308 into each of the additional image frames 310.

[0052] Traditional approaches to generating the additional image frames (or generally the plurality of output image frames as show n in Figure 2) suffer from high power consumption, high memory bandwidth requirement, and large silicon area. For example, in one traditional approach, the image processor can include a plurality of image transformation engines, where each image transformation engine generates a respective one of the plurality of output image frames in parallel from a set of input images. As an example, such a traditional approach could employ n image transformation engines to generate n output image frames based on n image transformation operations. Each of the n image transformation engines can include a local cache that stores a portion of each of the set of input images read from memory based on the portion of the respective output image being generated. For example, when generating the top left comer pixel of an output image frame, the corresponding image transformation engine can determine the one or more pixels of the set of input images needed to generate the top left comer pixel and read from memory portions of the set of input images that include the one or more pixels. The portions of the set of input images read from memory can be stored in a local cache to take advantage of data localization. While this approach can generate the output image frames with high throughput, additional silicon area is needed to accommodate the multiple image transformation engines and the associated caches. Further, each of the multiple image transformation engines will have its own separate memory read operation to read the portion of the set of input images. Therefore, the power consumed in reading the set of input images can be high.

[0053] In another traditional approach, a single image transformation engine may be employed, and the image transformation engine can generate each output image frame Atorney Docket No.: 56113-0781WO1 sequentially (i.e.. one after the other). That is, the image transformation engine generates an entire output image frame before generating the next output image frame. While this approach reduces the silicon area needed, the approach suffers from low throughput. In addition, the power consumed in reading the set of input images from memory is substantially the same as the previously discussed approach. Furthermore, this approach reads the input image data multiple times to generate the multiple output image frames, rendering the approach inefficient in terms of latency and power consumption.

[0054] In contrast with the above discussed approaches, the techniques discussed herein utilize a single image transformation engine 120 but do not suffer from the low throughput and high memory' read power consumption of the above-described traditional approaches. In particular, the approaches discussed herein generate output image frames in a manner (as further described below with reference to FIGS. 4-8) such that the portion of the image data for the input image stored in cache is utilized for the generation of similarly located portions of the output image frames. This results in high hit-rate in the cache for the input image data and reduces the need to access memory repeatedly for each output image frame generation.

[0055] Figure 4 shows an example sequence of output image frame generation by the image transformation engine. In particular, Figure 4 shows an example sequence of the generation of the plurality of output image frames 206 (OIF-1 to OIF-n) referred to previously in relation to Figure 2. The image transformation engine 120 generates a first output image frame portion 402 of the first output image frame OIF-1. To generate this portion in the output image frame, the image transformation engine 120 can load image data related to the set of input images 202 from memory' and into the cache 208. This image data can correspond to a subset of all the pixel data that form the input image. For example, the image data can correspond to one or more rows of pixels of the set of input images. The specific image data that is read form memory can be a function, in part, of the corresponding image transformation operation Op-1. Based on the corresponding image transformation operation 204, the image transformation engine 120 can transform appropriate portions of the image data of the input image into the first output image frame portion 402 of the first output image frame OIF-1. The image transformation engine 120 can then generate the same portion (the first output image frame portion 402) of the second output image frame OIF-2. As the same portion of the second output image frame OIF-2 is being generated, it is highly likely that the input image data needed to generate this portion of the output image frame is already present in the cache. As a result, the image transformation engine 120 can access the input image data from the cache without having to carry out a memory' read operation to the main Atorney Docket No.: 56113-0781WO1 memory. The image transformation engine 120 can continue to generate the first output image frame portion 402 of the subsequent output image frame with high likelihood that the input image data needed to generate those portions is present in the cache. Once the first output image frame portion 402 of all the plurality of output image frames is generated, the image transformation engine 120 can begin generating a second output image frame portion 404 of the plurality of output image frames. Again, due to localization of data, the image transformation engine 120 may have to access the input image data from main memory for generating the second output image portion 404 of only the first output image frame OIF-1. In this manner, the approach of image transformation discussed herein can utilize only a single image transformation engine while at the same time reduce the power consumption of memory read operations.

[0056] Figure 5 depicts a block diagram of an example image transformation engine 120. The image transformation engine 120 can include an output generation unit 502 coupled with a local cache 504. An axis mapper 506 utilizes image transformation operations to generate corresponding locations on the input image frame that are to be used to generate an output image frame portion. The axis mapper 506 can also provide to the output generation unit 502 a current location, on the output image frame, of the output image frame portion being generated. A sequencer 506 can generate the current location e.g., in x-axis and y-axis coordinates on the reference frame of the output image frame, and an identifier corresponding to each image transformation operation and a corresponding output image frame. A multiplexer 510 can select one of two image transformation operations: a function-based image transformation operation 512 and a grid-based image transformation operation 514. The image transformation engine 120 also can include a memory read unit (RDMA) 516 that receives an address of the memory location where the desired image data for the input image is stored in memory, reads the desired image data, and provides the desired image data to the output generation unit 502 and the local cache 504. The image data may be stored in an input image buffer 518 within the SoC (e.g., the SoC 140 shown in Figure 1) or in a main memory unit (e.g., the memory device 170 shown in Figure 1). A memory write unit (WDMA) 520 can receive a generated output image frame portion, a location of the portion within the output image frame and an identifier identifying the output image frame to which the generated portion belongs. The memory' w rite unit 520 can write to a plurality7of output image frame buffers 522-1 to 522-n. In particular, each output image frame buffer can correspond to an output image frame to be generated by the image transformation engine 120. Atorney Docket No.: 56113-0781WO1

[0057] Figure 6 shows a flow diagram of a process 600 for generating a plurality of output image frames from a set of input image frames. The process 600 can be executed, for example, by the image transformation engine 120 discussed herein.

[0058] The process 600 includes receiving a request to perform a plurality of image transformation operations on an input image to generate a plurality of output image frames (602). The image transformation engine 120 can receive requests for generating a plurality of output images from one or more components of the SoC 140. For example, the CPU 140 can be executing operations that need generation of output images from a set of input images. Such operations can include image warping, image re-sizing, lens distortion adjustment, frame rate modification, 2D to 3D conversion, etc. The CPU 140 can request the image transformation engine 120 to generate the output images. The request may also include the location in memory where the input image is stored. The request may also include location of where the image transformation operations are stored. The image transformation operations can indicate the type of operation the image transformation engine 120 has to perform to generate the plurality of output images.

[0059] The process 600 can also include assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated (604). The sequencer 508 can assign identifiers to image transformation operations, where each image transformation operation can correspond to an output image frame. The image transformation engine 120 can access from memory the plurality of image transformation operations and the sequencer 508 can assign identifiers to each of the image transformation operations. The image transformation operations can include, e.g., functionbased transformation operations 512 or grid-based transformation operations. The functionbased transformation operation can include an analytical function that describes the operation. An example operation can be pixel displacement. That is, each output image pixel at a specific location on the output image frame is generated based on an input image pixel whose location is determined based on a displacement (along the x and y axes) from the same specific location on the input image. For example, a function-based transformation operation can specify to displacement in the x-direction in the form of an equation: 2x + 2 and the displacement in y-direction in the form of an equation: 1.5y + 4, where ‘x’ and ‘y’ are the coordinates of the output image frame portion.

[0060] The grid-based transformation operation 514 can provide the displacement values based on a grid of points that are superimposed on the output image frame. Each point in the grid has associated displacement values (dx and dy). Based on the location (x, y) of the Atorney Docket No.: 56113-0781WO1 output image frame portion and the displacement values associated with one or more points on the grid near the location, the grid-based transformation operation can provide displacement values (dx and dy). Additional details of how the displacement values are determined based on the grid are discussed further below.

[0061] In some examples, the image transformation operation can be a function of the location (x. y) of the output image frame portion. That is, for example, the displacement (dx, dy) can be different for different locations in the output image frame. In some such instances, transformation operations can be stored in memory or a buffer and can be accessed by image transformation engine 120 based on the current location of the output image frame portion. In some instances, such as for function-based transformation operations, the same function can be applied to all portions within a single output image frame. In such instances, the displacement may not change based on the location of the output image frame portion, but may change from one output image frame to another.

[0062] The sequencer 508 can assign the correct identifier to the transformation operation such that the axis mapper 506 receives a data structure that include the displacement (or other transformation) information such as (dx, dy) and the ID indicating the transformation is to be applied to generate an output image frame associated with the ID. In some examples, the transformation operations can be accessed (or provided to the axis mapper 506) in a sequence that matches the sequence in which the output image frame portions are to be generated. The sequencer 508 can generate identifiers in an order that matches the sequence, such that the identifier correctly matches the transformation operation. For example, referring to Figure 4, a first portion of the sequence can include the transformation operations for the n output image frames associated with the first output image frame portion 402. The sequencer 508 can match this sequence and generate identifiers ID-1, ID-2, . . . . ID- / ? such that the first identifier ID-1 is assigned to the transformation operation to generate the first output image fame portion 402 for the first output image frame (OIF-1), the second identifier ID-2 is assigned to the transformation operation to generate the first output image frame portion 402 for the second output image frame (OIF-2), and so on, until identifiers are assigned to all transformation operations for remainder of the output image frames. A second portion of the sequence can include the transformation operations for the n output image frames associated with the second output image frame portion 504. Similarly other portions of the sequence can include the transformation operations for the n output image frames associated with other output image frame portions. The sequencer 508 can generate the identifiers to match these Atorney Docket No.: 56113-0781WO1 portions of the sequence such that the identifier correctly matches the output image frame being generated by the transformation operation.

[0063] It should be noted that the output image frame portion can be a single pixel or a group of two or more pixels that can be represented by a location on the output image frame. For example, the output image frame portion can be a single pixel that can be represented by the x and y coordinates in the reference frame of the output image frame. In some other examples, the output image frame portion can be an p pixel by q pixel portion of the output image frame, and the x and y coordinates can represent the location of one or more pixels within the p x q portion. In some instance, the location can be of a pixel at one comer of the p q portion. In other instances, the location can be of a pixel at the center of the p x q portion. Once the location of the representative pixel is known, then the locations of each pixel of the / ? x q portion of the output image frame can be determined.

[0064] The process 600 can also include store image data for the input image, obtained from memory', in a local cache (606). The axis mapper 506 can determine a memory' address where the desired image data for the input image is stored. As mentioned above, the axis mapper 506 can receive from the image transformation operation blocks 512, 514, information about the location of the pixel (or a portion of the input image) based on which the output image frame portion is generated. For example, the axis mapper 506 can receive a displacement information and the x and y coordinates on the output image. The axis mapper 506 can determine the location (x’, y’) of the input pixels based on applying the displacement information to the x and y coordinates. For example, to determine the location of the input pixel on the x-axis, the axis mapper 506 can determine x’ = x + dx, and to determine the location of the input pixel on the y-axis, the axis mapper 506 can determine y’ = y + dy. As an example, suppose the x and y coordinates of the output image frame portion are 12 and 7. respectively, and the displacement information is dx = 3 and dy = -2. In such instances, the axis mapper 506 can determine the coordinates x’ and y’ to be 15 and 5, respectively. The x’ and y’ values are the coordinates of the input pixels on the input image.

[0065] The axis mapper 506 can determine a memory address based on the coordinates of the input pixel in the input image. Generally, the memory location of the input pixel can be a function of the memory layout and the manner in which the image is stored in memory (e g., tiled or raster scan). In some examples, where the image data of the input image is stored row-by-row (raster scan), one can assume that the input image buffer 518, where the input image is stored can have a base address, and a line stride corresponding to each line or row of the input image in bytes (e.g., line-stride bytes), and each pixel value being represented by b Atorney Docket No.: 56113-0781WO1 bytes. In such instances, the address (Loc(x’, y', ID)) of the input pixel can be equal to “base address + y' * line-stride bytes + b * x' .” Of course, this is only an example, and different implementations can utilize different functions to convert the coordinates x’ and y’ for the input pixel into a memory location where that input pixel is stored.

[0066] Once the memory' address of the input pixel is known, the image transformation engine 120 can use the RDMA 516 to access the input image buffer 518. In most instances, the image data retrieved in response to the memory address represents only a portion of the input image. For example, a memory read operation may read a number of bytes starting at or including the memory' address, where the number of bytes can be equal to the page size of the memory' system. The page size of the memory' system can generally be smaller than the size, in bytes, of the input image. Thus, the image data retrieved from the input image buffer 518 can have a size, in bytes, that is smaller than the size of the input image. Further, as the page size is generally larger than the data size of the requested input pixel, the image data can include pixel values of not only the requested input pixel but pixel values of neighboring input pixels as well. The size of the cache can determine the number of input pixel values that can be stored in the cache. A larger cache can store a larger portion of the input image and can result in a higher probability' that a requested input pixel is present in the cache. However, a larger cache can increase silicon area, increase power consumption and increase cache read latency. Therefore, there is a tradeoff between improved performance due to larger cache size and the corresponding impact on silicon area, power consumption and increased cache latency.

[0067] The image data read from the input image buffer 518 can be stored in the local cache 504. The local cache 504 can be a fully associative cache, a direct mapped cache, or any other type of cache. Regardless of the type of the local cache 504, the latency for accessing data from the local cache 504 is substantially less than the latency for accessing data from the input image buffer 518. This is because the local cache 504 is local to the image transformation engine 120, while the input image buffer 518 is relatively remote and has to be accessed via the memory read unit 516. Further, the local cache 504 can be configured to be searchable based on the memory address of the input pixels generated by the axis mapper 506. Thus, when the axis mapper 506 requests an input pixel from memory, the memory' read operation first checks whether the pixel data w ith the requested memory' address is present in the local cache 504. If there is a hit in the cache, then the memory read operation can return the input pixel from the cache and refrain from carrying out a read operation from the input image buffer 518. If, however, there is a cache miss, that is, the requested input pixel is not Atorney Docket No.: 56113-0781WO1 present in the cache, then the memory operation can proceed to access the requested input pixel from the input image buffer 518.

[0068] Referring again to Figure 6, the process 600 can also include generating, using a portion of image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality of output image frames to be generated, wherein each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame (608). The output generation unit 502 can generate the output image frame portion based, in part, on the input pixel read from the input image buffer 518 or from the local cache 504. The axis mapper 506 can provide to the output generation unit 502 the coordinates of the output image frame portion as well as the ID. Based on the image transformation to be carried out, the output generation unit 502 can generate the corresponding output mage frame portion. In instances where the transformation operation is a warp function, i.e., where the output pixel is generated based on an input pixel at a different location in the input image, then the output generation unit 502 can use the pixel values of the input pixel for the values of the output pixel. For example, the input pixel value is represented by an 8-bit number that represents the value of the input pixel. The output generation unit 502 can use the same 8-bit value for the output image frame portion. In some other instances, the transformation operation may include additional pixel manipulation. For example, the transformation operation, in addition to warping, may include reducing the intensity of the output pixel in relation to the input pixel. In such instances, the output generation unit 502 can assign a value to the output image frame portion that is less than that of the input pixel.

[0069] In some instances, the output generation unit 502 may not have the capability to perform the desired transformation. In such instances, the output generation unit 502 can communicate with other components of the SoC 140 to carry out the transformation. For example, the output generation unit 502 can communicate with one or more of the ISP 112, the HPU 114, the DSP 116, the GPU 118, and the image processing engine 126 to cany7out processing of the input image pixels to generate the output image frame portion.

[0070] The process 600 can further include storing in memory and within a separate container associated with each of the plurality of output image frames, the respective output image portions (610). The output generation unit can generate the output image frame portion along with the ID and provide the generated data to the memory write unit 520. The memory write unit can inspect the ID and determine which one of the plurality of containers to store the output image frame portion. A container can refer to memory allocated to store Atorney Docket No.: 56113-0781WO1 an output image frame. A plurality of containers can be utilized to store the plurality of output image frames. As each portion of an output image frame is generated, the portion can be stored in a respective container. As an example, the container can refer to an output image frame buffer to store the output image frame. The memory write unit is coupled with the plurality7of output image frame buffers, and each output image frame buffer stores data associated with an output image frame to be generated. In some examples, the memory write unit 520 can include an address translation unit that converts the coordinates x. y, associated with the output image frame portion into a memory address. The memory address can be a function of the manner in which the output image frame is stored in the output image frame buffer. For example, the memory address may be different based on whether the output image frame is stored in a tiled manner or in a raster-scan, row-by-row manner. In some examples, where the image data of the output image is stored row-by-row (raster scan), one can assume that the output image buffer, where the output image frame is stored, can have a base address, and a line stride corresponding to each line or row7of the output image in bytes (e.g., line-stride bytes), and each pixel value being represented by b bytes. In such instances, the address (Loc(x, y, ID)) of the output image frame portion can be equal to '‘base address + y * line-stride bytes + b * x.” Once the memory address is determined, the memory write unit 520 can determine which ones of the output image fame buffers have an associated ID that is same as the ID of the output image frame portion. The memory write unit 520 can then store the output image frame portion at the generated memory address in the appropriate output image frame buffer.

[0071] In some instances, the x’ and y’ coordinates of the input pixel generated by the axis mapper 506 may not match actual coordinates of the input pixels in the input image. For example, suppose the x and y coordinates of the output image frame portion is 12 and 7, respectively, and the displacement information is dx = 3.2 and dy = 2.5. The resulting coordinates on the input image frame will be x’ = 15.2 and y’= 9.5. These coordinates may not align exactly with any single input pixel in an input image. In such instances, the axis mapper 506 and the output generation unit can be configured to utilize several approaches.

[0072] In one example, the axis mapper 506 can be configured to determine the nearest input pixel to the resulting coordinates, and the output generation unit can generate the output image frame portion based on the value of the nearest input pixel. Referring to the example above, if the nearest pixel to the coordinates of x’ = 15.2 and y‘ = 9.5 is x’ = 15 and y’ = 9, then the axis mapper can generate the memory address for the nearest input pixel. The nearest pixel can be accessed from the local cache 504, if present, or from the input image Atorney Docket No.: 56113-0781WO1 buffer 518. The output generation unit 502 can then utilize the value of the nearest pixel to generate the output image frame portion. In another example, the axis mapper 506 can be configured to determine the nearest set of input pixels and interpolate the values of the set of input pixels to generate the output image frame portion.

[0073] Figure 7 shows an example approach to generating the output image frame portion based on input pixel values. In particular, Figure 7 shows a portion of the input image 700, where the portion includes four input pixels 702, 704. 706 and 708 nearest to a x‘ and y ’ coordinates 710 generated by the axis mapper 506. The x’ and y’ coordinates for an input pixel generated by the axis mapper 506 do not coincide with any of the pixels in the input image 700. In such instances, the output generation unit 502 can determine the value of the input pixel at the x’ and y’ coordinate by interpolation based on the values of the nearest four input pixels 702, 704, 706, and 708. In one example, the output generation unit 502 can determine the value of the input pixel at the x’ and y’ coordinates 710 based on bilinear interpolation. In some other examples, the output generation unit 502 can be configured to perform bicubic interpolation at the x’ and y ' coordinates 710. Bicubic interpolation ty pically utilizes values of the 16 nearest input pixels to the x’ and y’ coordinates 710. In such instances, the axis mapper 506 can be configured to determine the coordinates of the 16 input pixels in the input image, determine the memory addresses of the 16 input pixels, and request the values of these 16 input pixels from the local cache 504 or the input image buffer 518. Of course, bilinear interpolation and bicubic interpolation are only examples, and that the output generation unit 502 can be configured to utilize other interpolation techniques such as linear interpolation, polynomial interpolation, spline interpolation, etc.

[0074] As mentioned previously, the grid-based transformation operation 514 can provide displacement values based on a grid of points that are superimposed on the output image frame. Figure 8 shows an example grid-based transformation operation approach to determining displacement values for an output image frame portion. In particular, Figure 8 shows an output image frame 802 that the output generation unit 502 can generate. A grid 804 including a set of grid points 804 can be superimposed on the output image frame 802. The number of grid points 804 in the grid can be substantially less than the number of pixels in the output image frame 802. In some examples, the ratio between the number of pixels in the output image frame 802 to the number of grid points 804 can be between 1 to 1000. In the example shown in Figure 8, the grid includes 24 grid points (labeled G(0.0) to G(3,7)). Nevertheless, in some instances, the size of the grid can be the same as the size of the output Atorney Docket No.: 56113-0781WO1 image frame portion, such that each grid point represents the transformation operation for each corresponding pixel in the output image frame.

[0075] Each grid point 804 of the grid has an associated transformation operation value. For example, where the transformation operation specifies a displacement of the input pixel to generate the portion of the output image frame, each grid point 804 can have an associated displacement value. As an example, the displacement values for grid points G(l,2), G(l,3), G(2,2). and G(2.3) can be equal to {5.-2}. { 1.2}, {2,0}. and {-1,3}. respectively. These displacement values specify the location of the input pixel from the location of the output pixel by the displacement value.

[0076] The transformation operation value at each grid point 804 specifies the transformation operation value for the output pixel directly under the grid point 804 on the output image frame 803. In this manner, the grid specifies transformation operation values for only those output pixels that are positioned in a corresponding position on the output image frame 802. In examples where the number of grid points is less than the number of output pixels in the output image frame 802. there are output pixels that do not have a corresponding grid point in the grid at the same position as the output pixel. In such instances, the transformation operation value for those pixels can be determined based on approximations from the nearest grid points. In one example, an output pixel 806 is positioned on the output image frame 802 at a position (12, 7). No grid point is positioned over the position (12, 7). Therefore, the transformation operation value for the output pixel 806 can be determined based on transformation operation values of neighboring grid points. In one approach, the transformation operation value of the output pixel 806 can be equal to the transformation operation value of the nearest grid point G(l,2). In another approach, the transformation operation value of the output pixel 806 can be an average of the transformation operation value of the nearest four gird points (G(l,2), G(l,3), G(2,2), and G(2,3)). In yet another approach, interpolation can be used to determine the transformation operation value of the output pixel 806. For example, bilinear interpolation can be used to determine the transformation operation value for the output pixel 806 based on the transformation operation values of the nearest four grid points G(1.2), G(1.3), G(2.2), and G(2,3). Of course, other interpolation techniques such as linear interpolation, polynomial interpolation, spline interpolation, etc., can also be used.

[0077] Using grid-based transformation operation values can save memory. For generating the output image frame, transformation operation value for the output image frame portion at each x, y coordinate value may be needed. For large images with millions of pixels, storing Atorney Docket No.: 56113-0781WO1 the transformation operation values for each pixel may require a large amount of memory. By storing transformation operation values for only a subset of the coordinate locations can save valuable memory space. The transformation operation values for other coordinate locations can be determined based on one or more approximation techniques.

[0078] The generation of the output image is carried in a sequence to allow for utilization of already stored input image data in the local cache 504. In particular, the sequencer 508 specifies the sequence in which the output image frame portions of various output image frames are generated. In particular, the sequencer increments the identifier ID each cycle while maintaining the same x, y coordinates. Here, a cycle can refer to one clock cycle or a number of clock cycles needed for the output generation unit 502 to generate an output image frame portion. As partly depicted in Figure 4. the output generation unit 502 receives a transformation operation and the location of the output image frame portion for one output image frame first, followed by the transformation operation and the location of the output image frame portion of another output image frame, and so on until the output image frame portions at the same location for all output image frames has been generated. More specifically, the sequencer 508 can begin a first sequence associated with a first location x, y, and in this first sequence, the output generation unit 502 can generate output image frame portions located at the same position x, y, for each of the plurality of output image frames using the respective transformation operation. Subsequently, by changing the location x, y, to another value in a second sequence, the output generation unit 502 can generate output image frame portions at that new location x, y, for each of the plurality of output image frames based on the respective transformation operation. The output generation unit 502 can continue the sequence of operations for all of the desired locations x, y. of the output image frame portions.

[0079] The write memory unit 520 can store the output image frame portion in the appropriate output image frame buffer based on the identifier. In one approach, within each output image frame buffer, the write memory7unit 502 can store the output image frame portions serially as they are generated. For example, the write memory unit 502 can increment the memory address with each output image frame portion when storing the output image frame portions in an image frame buffer. If the output image frame buffer represents a pixel, then the write memory7unit 502 can write into the output image frame buffer pixel by pixel with each pixel being stored at the next appropriate memory address depending upon the size in bytes of the pixel. When all the output image frame portions are stored in the output image frame buffer, the output image frame can be considered to have been generated. Atorney Docket No.: 56113-0781WO1

[0080] The manner in which the output image frame is stored in the output image frame buffer can correspond to the manner in which the output image frame portions are generated. For example, referring to Figure 4, if the output image frame portions are formed row-by-row with the sequence show in Figure 4, then the output image frame in the output image frame buffer would also be stored row-by-row. A component (such as the image processing engine 126, the CPU 104, etc.) reading the image frame buffer could be pre-configured to know the base address in the output image buffer where the image data begins, the line stride in bytes, and the size in bytes used to store the value of each pixel. Using this information the component can form the output image frame and / or cany7out additional processing as needed.

[0081] The techniques discussed herein generate multiple output image frames by transforming a set of input images. Each output image frame is generated by generating a plurality of output image frame portions where similarly located output image frame portions belonging to multiple output image frames are generated in sequence such that input image frame data can be reused from a local cache to reduce the number of read requests to a main memory where the input image data is stored. The techniques discussed herein at the least reduce the power consumption by reducing the number of main memory read operations.

[0082] The image transformation engine 120 as well as the processes discussed herein can be implemented on a computing system. In particular, a computing system including a computing device and / or a mobile computing device can be used to implement the techniques described herein. For example, one or more processes, electronic design tools, and data can be implemented on or stored in the computing device or the mobile computing device.

[0083] The computing device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0084] The computing device includes a processor, a memory', a storage device, a high-speed interface connecting to the memory and multiple high-speed expansion ports, and a low- speed interface connecting to a low-speed expansion port and the storage device. Each of the processor, the memory, the storage device, the high-speed interface, the high-speed Atorney Docket No.: 56113-0781WO1 expansion ports, and the low-speed interface, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor can process instructions for execution within the computing device, including instructions stored in the memory or on the storage device to display graphical information for a GUI on an external input / output device, such as a display coupled to the high-speed interface. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor is a single threaded processor. In some implementations, the processor is a multi-threaded processor. In some implementations, the processor is a quantum computer.

[0085] The memory stores information within the computing device. In some implementations, the memory is a volatile memory7unit or units. In some implementations, the memory is a non-volatile memory unit or units. The memory may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0086] The storage device is capable of providing mass storage for the computing device. In some implementations, the storage device may be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine- readable mediums (for example, the memory, the storage device, or memory on the processor). The high-speed interface manages bandwidth-intensive operations for the computing device, while the low-speed interface manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface is coupled to the memory7, the display (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports, which may accept various expansion cards (not shown). In the implementation, the low-speed interface is coupled to the storage device and the low-speed expansion port. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be Atorney Docket No.: 56113-0781WO1 coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g.. through a network adapter.

[0087] The computing device may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer. It may also be implemented as part of a rack server system. Alternatively, components from the computing device may be combined with other components in a mobile device, such as a mobile computing device. Each of such devices may include one or more of the computing device and the mobile computing device, and an entire system may be made up of multiple computing devices communicating with each other.

[0088] The mobile computing device includes a processor, a memory, an input / output device such as a display, a communication interface, and a transceiver, among other components. The mobile computing device may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor, the memory, the display, the communication interface, and the transceiver, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0089] The processor can execute instructions within the mobile computing device, including instructions stored in the memory. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the mobile computing device , such as control of user interfaces, applications run by the mobile computing device , and wireless communication by the mobile computing device .

[0090] The processor may communicate with a user through a control interface and a display interface coupled to the display. The display may be, for example, a TFT (Thin-Film- Transistor Liquid Crystal Display) display or an OLED (Organic Light Emiting Diode) display, or other appropriate display technology. The display interface may include appropriate circuitry for driving the display to present graphical and other information to a user. The control interface may receive commands from a user and convert them for submission to the processor. In addition, an external interface may provide communication with the processor, so as to enable near area communication of the mobile computing device with other devices. The external interface may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used. Atorney Docket No.: 56113-0781WO1

[0091] The memory stores information within the mobile computing device. The memory can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory may also be provided and connected to the mobile computing device through an expansion interface, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory may provide extra storage space for the mobile computing device, or may also store applications or other information for the mobile computing device. Specifically, the expansion memory may include instructions to cany' out or supplement the processes described herein and may include secure information also. Thus, for example, the expansion memory may be provided as a security module for the mobile computing device, and may be programmed with instructions that permit secure use of the mobile computing device. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non- hackable manner.

[0092] The memory may include, for example, flash memory and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory, the expansion memory, or memory on the processor). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver or the external interface.

[0093] The mobile computing device may communicate wirelessly through the communication interface, which may include digital signal processing circuitry' in some cases. The communication interface may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 4G / 5G / 6G cellular, among others. Such communication may occur, for example, through the transceiver using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS Atorney Docket No.: 56113-0781WO1

[0094] (Global Positioning System) receiver module may provide additional navigation- and location-related wireless data to the mobile computing device, which may be used as appropriate by applications running on the mobile computing device.

[0095] The mobile computing device may also communicate audibly using an audio codec, which may receive spoken information from a user and convert it to usable digital information. The audio codec may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device.

[0096] The mobile computing device may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.

[0097] Embodiments of the subject mater and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible nontransitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0098] The term “data processing apparatus7’ refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (applicationspecific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution Atorney Docket No.: 56113-0781WO1 environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. -

[0099] A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0100] The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a readonly memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0101] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory Atorney Docket No.: 56113-0781WO1 devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. -While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0102] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0103] Particular embodiments of the subject mater have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0104] What is claimed is:

Claims

1. Atorney Docket No.: 56113-0781WO1CLAIMS1. A method, comprising: receiving a request to perform a plurality of image transformation operations on an input image to generate a plurality of output image frames; assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated; storing image data for the input image, obtained from memory, in a local cache; generating, using a portion of the image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality of output image frames to be generated, wherein each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame; and storing, in memory and within a separate container associated with each of the plurality of output image frames, the respective output image frame portions.

2. The method of claim 1 , wherein the respective output image frame portion includes an output pixel, the method further comprising: determining location of one or more input pixels in the input image based on a location of the output pixel in the respective output image frame and the respective image transformation operation; reading from the local cache the one or more input pixels based on the location of the one or more input pixels; and generating the output pixel based on the one or more input pixels.

3. The method of claim 2, further comprising: generating an output pixel data structure including a value of the output pixel and the respective identifier corresponding to the respective output image frame on which the output pixel is located; and storing the value of the output pixel within the separate container associated with the respective output image frame.

4. The method of claim 1 , wherein the image transformation operation includes a displacement value, and a location of the portion of the image data on the input image isAtorney Docket No.: 56113-0781WO1 determined based on a location of output image frame portion on the respective output image frame and the displacement value.

5. The method of claim 4, wherein the location of the portion of the image data on the input image includes a plurality7of input pixels on the input image, the method further comprising determining a value for an output pixel in the respective output image frame portion based on interpolating values of the plurality of input pixels.

6. The method of claim 1 , further comprising generating output image frame portions of the plurality of output image frames in successive single clock cycles.

7. The method of claim 1, wherein at least one image transformation operation of the plurality of image transformation operations includes a displacement value determined based on displacement values at a set of grid points located at positions adjacent to a position of the respective output image frame portion.

8. The method of claim 1, further comprising: generating the respective output image frame portions located at a same position in their respective plurality of output image frames in a first sequence, and generating another set of respective output image frame portions located at another same position in the respective plurality of output image frames in a second sequence subsequent to the first sequence.

9. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to perform operations for determining buffering components for placement between two nodes on an integrated circuit design, the operations comprising: receiving a request to perform a plurality7of image transformation operations on an input image to generate a plurality of output image frames; assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated; storing image data for the input image, obtained from memory7, in a local cache; generating, using a portion of the image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality ofAtorney Docket No.: 56113-0781WO1 output image frames to be generated, wherein each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame; storing, in memory and within a separate container associated with each of the plurality7of output image frames, the respective output image frame portions.

10. The computer-readable storage media of claim 9, wherein the respective output image frame portion includes an output pixel, the operations further comprising: determining location of one or more input pixels in the input image based on a location of the output pixel in the respective output image frame and the respective image transformation operation; reading from the local cache the one or more input pixels based on the location of the one or more input pixels; and generating the output pixel based on the one or more input pixels.

11. The computer-readable storage media of claim 10, the operations further comprising: generating an output pixel data structure including a value of the output pixel and the respective identifier corresponding to the respective output image frame on which the output pixel is located; and storing the value of the output pixel within the separate container associated with the respective output image frame.

12. The computer-readable storage media of claim 9, wherein the image transformation operation includes a displacement value, and a location of the portion of the image data on the input image is determined based on a location of output image frame portion on the respective output image frame and the displacement value.

13. The computer-readable storage media of claim 12, wherein the location of the portion of the image data on the input image includes a plurality of input pixels on the input image, the operations further comprising determining a value for an output pixel in the respective output image frame portion based on interpolating values of the plurality7of input pixels.Atorney Docket No.: 56113-0781WO114. The computer-readable storage media of claim 9, the operations further comprising generating output image frame portions of the plurality of output image frames in successive single clock cycles.

15. The computer-readable storage media of claim 9, wherein at least one image transformation operation of the plurality of image transformation operations includes a displacement value determined based on displacement values at a set of grid points located at positions adjacent to a position of the respective output image frame portion.

16. The computer-readable storage media of claim 9, the operations further comprising: generating the respective output image frame portions located at a same position in their respective plurality of output image frames in a first sequence, and generating another set of respective output image frame portions located at another same position in the respective plurality of output image frames in a second sequence subsequent to the first sequence.

17. A system, comprising: one or more processors; and one or more storage devices storing instructions that when executed by the one or more processors to perform operations for determining buffering components for placement between two nodes on an integrated circuit design, the operations comprising: receiving a request to perform a plurality' of image transformation operations on an input image to generate a plurality of output image frames; assigning an identifier corresponding to each respective image transformation operation and a corresponding output image frame to be generated; storing image data for the input image, obtained from memory', in a local cache; generating, using a portion of the image data for the input image stored in the local cache, a respective output image frame portion corresponding to each of the plurality of output image frames to be generated, wherein each output image frame portion is generated using an image transformation operation corresponding to the identifier for the respective output image frame; and storing, in memory and within a separate container associated with each of the plurality of output image frames, the respective output image frame portions.Atorney Docket No.: 56113-0781WO118. The system of claim 17, wherein the respective output image frame portion includes an output pixel, the operations further comprising: determining location of one or more input pixels in the input image based on a location of the output pixel in the respective output image frame and the respective image transformation operation; reading from the local cache the one or more input pixels based on the location of the one or more input pixels; and generating the output pixel based on the one or more input pixels.

19. The system of claim 18, the operations further comprising: generating an output pixel data structure including a value of the output pixel and the respective identifier corresponding to the respective output image frame on which the output pixel is located; and storing the value of the output pixel within the separate container associated with the respective output image frame.

20. The system of claim 17, wherein the image transformation operation includes a displacement value, and a location of the portion of the image data on the input image is determined based on a location of output image frame portion on the respective output image frame and the displacement value.

21. The system of claim 20, wherein the location of the portion of the image data on the input image includes a plurality of input pixels on the input image, the operations further comprising determining a value for an output pixel in the respective output image frame portion based on interpolating values of the plurality of input pixels.

22. The system of claim 17, the operations further comprising generating output image frame portions of the plurality of output image frames in successive single clock cycles.

23. The system of claim 17, wherein at least one image transformation operation of the plurality of image transformation operations includes a displacement value determined based on displacement values at a set of grid points located at positions adjacent to a position of the respective output image frame portion.Attorney Docket No.: 56113-0781WO124. The system of claim 17, the operations further comprising: generating the respective output image frame portions located at a same position in their respective plurality of output image frames in a first sequence, and generating another set of respective output image frame portions located at another same position in the respective plurality of output image frames in a second sequence subsequent to the first sequence.

Images