COMPILER-MANAGED MEMORY FOR IMAGE PROCESSOR

Abstract

Image processor comprehensive: a two-dimensional execution path matrix (305; 405); a two-dimensional shift register matrix (306; 406), wherein the two-dimensional execution path matrix (305; 405) has a smaller dimension than the two-dimensional shift register matrix (306; 406) and wherein the two-dimensional shift register matrix (306; 406) comprises the following: a region that is coextensive with the two-dimensional execution path matrix (305; 405); and a ring area (409) of registers that is peripheral to the two-dimensional execution path matrix (305; 405); a memory (307; 407) that is locally coupled to the two-dimensional execution path matrix (305; 405) and the two-dimensional shift register matrix (306; 406), a sheet generator for loading a number of image data sheets into memory (307; 407), wherein each image data sheet of the number of image data sheets has the same dimensions as the two-dimensional execution path matrix (305; 405); The image processor is configured to execute program code, the program code causing the image processor to execute a procedure that includes the following: Loading a first image data sheet (701) of the number of image data sheets from the sheet generator into memory (307; 407); Loading a second image data sheet of the number of image data sheets from the sheet generator into memory (307; 407); Loading the first image data sheet (701) from memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406); Loading a portion of the second image data sheet (702) from memory (307; 407) into the ring area (409), wherein the image content of the second image data sheet (702) adjoins the image content of the first image data sheet (701); and Determining output values ​​for the first image data sheet (701) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​involves shifting image data of the first image data sheet (701) and the area of ​​the second image data sheet (702) within the two-dimensional shift register matrix (306; 406), wherein a stencil size used in determining the output values ​​includes only image points that lie within the two-dimensional shift register matrix (306; 406).

Description

Field of the invention

[0001] The field of the invention relates generally to image processing and, more particularly, to a compiler-managed memory for an image processor. General state of the art

[0002] Image processing typically involves processing pixel values ​​arranged in a matrix. A spatially arranged two-dimensional matrix captures the two-dimensional nature of the images (additional dimensions may include time (e.g., a sequence of two-dimensional images) and data type (e.g., colors). In a typical scenario, the ordered pixel values ​​are provided by a camera that has captured a still image or a sequence of frames to capture motion images. Conventional image processors typically fall into one of two extremes.

[0003] A first extreme performs image processing steps as software programs running on a general-purpose processor or a general-purpose processor (e.g., a general-purpose processor with vector instruction extensions). Although the first extreme typically provides a versatile application software development platform, its use of finer-grained data structures combined with the associated overhead (e.g., instruction fetching and decoding, handling of on-chip and off-chip data, speculative execution) ultimately results in the consumption of larger amounts of energy per unit of data during program code execution.

[0004] A second, opposite extreme applies stationary, hard-wired circuits to much larger data blocks. The use of larger (as opposed to finer) data blocks applied directly to user-defined circuits significantly reduces energy consumption per unit of data. However, the use of user-defined stationary functional circuits generally results in a limited set of operations that the processor can perform. Accordingly, the second extreme lacks the versatile programming environment (associated with the first extreme).

[0005] A technology platform that offers both versatile application software development capabilities and improved energy efficiency per data unit remains a desirable yet missing solution.

[0006] US 2015 / 0 086 134 A1 relates to a convolutional image processor comprising a loading and storing unit, a shift register unit and a mapping unit.

[0007] WO 2016 / 171 926 A1 relates to a device comprising an execution trace array coupled to a two-dimensional shift register array structure.

[0008] Hegarty, James et al.: Darkroom: Compiling High-Level Image Processing Code into Hardware Pipelines. ACM Trans. Graph., 2014, Vol. 33, No. 4, pp. 144:1–114:11, refers to Darkroom, a language and compiler for image processing. Brief description

[0009] An image processor according to claims 1 to 8, a computer system according to claim 9, a computer program product according to claims 10 to 15, a machine-readable storage medium according to claim 16 and a method according to claims 17 to 22 are disclosed. List of characters

[0010] The following description and accompanying drawings serve to illustrate embodiments of the invention. In the drawings: shows Fig. 1 an embodiment of an image processor hardware architecture; The Fig. 2a, Fig. 2b, Fig. 2c, Fig. 2d and Fig. 2e shows the parsing of image data into a row group, the parsing of a row group into a sheet, and the operation performed on a sheet with overlapping templates; Fig. 3a shows an embodiment of a stencil processor; Fig. 3b shows an embodiment of a command word of the template processor; Fig. 4 shows an embodiment of a data calculation unit within a template processor; The Fig. 5a, Fig. 5b, Fig. 5c, Fig. 5d, Fig. 5e, Fig. 5f, Fig. 5g, Fig. 5 hours, Fig. 5i, Fig. 5 years and Fig. 5k shows an example of using a two-dimensional displacement matrix and an execution trajectory matrix to determine a pair of adjacent output pixel values ​​with overlapping templates; Fig. 6 shows an embodiment of a unit cell for an integrated execution trajectory matrix and a two-dimensional displacement matrix; The Fig. 7a-7c relate to a compiler-managed memory access approach for an image processor data computation unit; Fig. 8 shows a memory access method for an image processor; The Fig. 9a and Fig. 9b concerns an instruction used to generate a memory address; Fig. 10 concerns another compiler-managed memory access approach; Fig. 11 shows an embodiment of a computer system. Detailed description of image processor hardware architecture and operation

[0011] Fig. 1 shows an embodiment of an architecture 100 for a hardware-implemented image processor. The image processor can be controlled, for example, by a compiler that converts program code written for a virtual processor in a simulated environment into program code that is actually executed by the hardware processor. As shown in Fig. 1, the architecture 100 includes a plurality of line buffer units 101_1 through 101_M connected to a plurality of stencil processor units 102_1 through 102_N and corresponding leaf generator units 103_1 through 103_N via a network 104 (e.g., a network on chip (NOC), including, but not limited to, a chip switch network, an on-chip ring network, or another type of network). In one embodiment, each line buffer unit may be connected to each leaf generator and a corresponding stencil processor via the network 104.

[0012] In one embodiment, the program code is compiled and loaded onto a corresponding stencil processor 102 to execute the image processing operations previously defined by a software developer (the program code may also be loaded onto the associated sheet generator of the stencil processor 103, depending on the design and implementation). In at least some cases, an image processing pipeline may be implemented by loading a first core program for a first pipeline phase into a first stencil processor 102_1, a second core program for a second pipeline phase into a second stencil processor 102_2, etc., with the first core system performing the functions of the first pipeline phase, the second core system performing the functions of the second pipeline phase, etc., and installing additional control flow methods to route output image data from one pipeline phase to the next pipeline phase.

[0013] In other configurations, the image processor may be implemented as a parallel machine comprising two or more template processors 102_1, 102_2 running on the same core program code. For example, a highly dense and high-rate stream of image data may be processed by distributing frames across multiple template processors, each performing the same function.

[0014] In still other configurations, essentially any DAG of core systems can be loaded onto the hardware processor by configuring respective template processors with their own respective core system program codes and configuring appropriate control flow hooks into the hardware to direct output images from one core system to the input of a next core system in the DAG construct.

[0015] In a general flow, frames of image data are received by a macro I / O unit 105 and transferred to one or more of the line buffer units 101 on a frame-by-frame basis. A particular line buffer unit parses its frame of image data into a smaller area of ​​image data called a "line group," and then passes the line group through the network 104 to a particular sheet generator. A complete or "full" singular line group may, for example, be composed of the data from several contiguous complete rows or columns of a frame (for simplicity, this description primarily refers to contiguous rows). The sheet generator further parses the line group of image data into a smaller area of ​​image data called a "sheet," and presents the sheet to its corresponding template processor.

[0016] In the case of a single-input image processing pipeline or DAG flow, input frames are generally routed to the same line buffer unit 101_1, which parses the image data into line groups and routes the line groups to the leaf generator 103_1, whose corresponding stencil processor 102_1 executes the code of the first core system in the pipeline / DAG. After the stencil processor 102_1 completes its operations on the line groups it processed, the leaf generator 103_1 sends output line groups to a "downstream" line buffer unit 101_2 (in some use cases, the output line group may be sent back to the same line buffer unit 101_1 that previously sent the input line groups).

[0017] One or more "receiver core systems," representing the next phase / operation in the pipeline / DAG, executed on their own respective other leaf generator and stencil processor (e.g., leaf generator 103_2 and stencil processor 102_2), then receive the image data generated by the first stencil processor 102_1 from the downstream line buffer unit 101_2. In this way, the output data of a "producer core system" running on a first stencil processor is forwarded to a "receiver core system" running on a second stencil processor, with the receiver core system executing the next set of operations after the producer core system according to the design of the overall pipeline or DAG.

[0018] A stencil processor 102 is configured to simultaneously operate on multiple overlapping stencils of image data. The multiple overlapping stencils and the internal hardware processing capacity of the stencil processor effectively determine the size of a sheet. Here, within a stencil processor 102, matrices of execution lanes cooperate to simultaneously process the image data surface area covered by the multiple overlapping stencils.

[0019] As described in more detail below, in various embodiments, leaves of image data are loaded into a two-dimensional register matrix structure within stencil processor 102. The use of leaves and the two-dimensional register matrix structure is believed to provide effective power improvements by moving a large amount of data into a large register area. For example, a single load operation with processing steps performed directly on the data is immediately followed by an execution lane matrix. Furthermore, the use of an execution lane matrix and a corresponding register matrix provides various stencil sizes that are easily programmable / configurable.

[0020] The Fig. 2a to Fig. 2e comprehensively illustrate embodiments of both the parsing activity of a line buffer unit 101 and the finer parsing activity of a sheet generator unit 103, as well as the stencil processing activity of the stencil processor 102 coupled to the sheet generator unit 103.

[0021] Fig. 2a shows an embodiment of an input frame of the image data 201. Fig. Figure 2a also shows an outline of three overlapping stencils 202 (each having a dimension of 3 pixels × 3 pixels) for which a stencil processor is designed. The output pixel for which each stencil generates the output image data is highlighted in black. For simplicity, the three overlapping stencils 202 are shown overlapping only in the vertical direction. It is relevant to recognize that a stencil processor may in reality be designed to have stencils overlapping in both the vertical and horizontal directions.

[0022] Due to the vertical overlapping stencils 202 within the stencil processor, as shown in Fig. 2a, there is a wide band of image data within the frame that a single stencil processor can operate. As described in more detail below, in one embodiment, the stencil processors within their overlapping stencils process data from left to right across all of the image data (and then repeat the process for the next group of lines in top-to-bottom order). Thus, as the stencil processors continue their operation, the number of black output pixel blocks increases horizontally to the right. As mentioned above, a line buffer unit 101 is responsible for parsing a line group of input image data from an incoming frame sufficient for the stencil processors to operate an extended number of pending cycles. An exemplary representation of a line group is shown as shaded area 203.In one embodiment, the line buffer unit 101 may include different dynamics for sending / receiving a line group to / from a leaf generator. For example, according to one mode referred to as "full group," the entire full-width image data lines are transmitted between a line buffer unit and a leaf generator. According to a second mode referred to as "virtual large," a line group is first transmitted with a subset of full-width lines. The remaining lines are then transmitted sequentially in smaller chunks (less than full width).

[0023] Once the row set 203 of input image data has been defined by the row buffer unit and passed to the leaf generator unit, the leaf generator unit further parses the row set into finer leaves that more precisely match the hardware limitations of the stencil processor. Specifically, in one embodiment, as described in more detail below, each stencil processor is formed from a two-dimensional shift register array. The two-dimensional shift register array essentially shifts image data "beneath" a matrix of execution lanes, with the pattern of shifting causing each execution lane to operate on data within its own respective stencil (i.e., each execution lane processes its own stencil of information to generate output for that stencil).In one embodiment, leaves are surface areas of input image data that "fill" or are otherwise loaded into the two-dimensional shift register matrix.

[0024] As described in more detail below, in various embodiments, there are actually multiple layers of two-dimensional register data that can be shifted on each cycle. For simplicity, much of this description simply uses the term "two-dimensional shift register" and the like to refer to structures that have one or more of these layers of two-dimensional register data that can be shifted.

[0025] As in Fig. 2b, the sheet generator parses a starting sheet 204 from the row group 203 and makes it available to the template processor (here, the data sheet corresponds to the shaded area, generally identified by the reference number 204). As shown in the Fig. 2c and Fig. 2d, the template processor operates on the sheet of input image data by effectively moving the overlapping templates 202 from left to right across the sheet. As shown in Fig. 2d, the number of pixels for which an output value could be calculated from the data within the sheet is exhausted (no other pixel positions can have an output value determined from the information within the sheet). For simplicity, the edge regions of the image have been ignored.

[0026] As in Fig. 2e, the sheet generator then provides a next sheet 205 for the template processor to continue operations. Note that when the templates begin operating on the next sheet, their initial positions correspond to the next progression (as previously described in Fig. 2d) from the exhaustion point to the right on the first sheet. With the new sheet 205, the stencils simply continue to move to the right, while the stencil processor operates on the new sheet in the same way as it did when processing the first sheet.

[0027] Note that there is some overlap between the data of the first sheet 204 and the data of the second sheet 205 due to the edge regions of the stencils surrounding an output pixel location. This overlap could be easily handled by the sheet generator transmitting the overlapped data twice. In alternative implementations, to feed a next sheet to the stencil processor, the sheet generator can continue to send only new data to the stencil processor while the stencil processor uses the overlapped data from the previous sheet. b. Stencil processor design and operation

[0028] Fig. 3a shows an embodiment of a template processor architecture 300. As in Fig. As shown in Figure 3a, the stencil processor includes a data calculation unit 301, a scalar processor 302 and associated memory 303, and an I / O unit 304. The data calculation unit 301 includes a matrix of execution lanes 305, a two-dimensional shift matrix structure 306, and separate random access memories 307 associated with specific rows or columns of the matrix.

[0029] The I / O unit 304 is responsible for loading "input" data sheets received from the sheet generator into the data calculation unit 301, as well as storing the sheets "output" from the template processor in the sheet generator. In one embodiment, loading sheet data into the data calculation unit 301 comprises parsing a received sheet into the rows / columns of image data and loading the rows / columns of image data into the two-dimensional shift register structure 306 or into the respective row / column random access memories 307 of the execution lane matrix (as described in more detail below). Once the sheet is initially loaded into the memories 307, the individual execution lanes within the execution lane matrix 305 can then load the leaf data, as appropriate (e.g., as a load instruction just before operating on the leaf data), into the two-dimensional shift register structure 306 of the random access memories 307.Upon completion of loading a data sheet into the register structure 306 (whether directly from a sheet generator or from the memories 307), the execution lanes of the execution lane matrix 305 operate on the data and ultimately "write" the completed data as a sheet directly "back" to the sheet generator or to the random access memories 307. In the latter case, the I / O unit 304 retrieves the data from the random access memories 307 to form an output sheet, which is then passed to the sheet generator.

[0030] Scalar processor 302 includes a program controller 309 that reads the stencil processor program code instructions from scalar memory 303 and issues the instructions to the execution lanes in execution lane matrix 305. In one embodiment, a single instruction is transferred to all execution lanes within matrix 305 to effect SIMD-like behavior of data calculation unit 301. In one embodiment, the instruction format of the instructions read from scalar memory 303 and issued to the execution lanes of execution lane matrix 305 includes a very long instruction word (VLIW) format, which includes more than one opcode per instruction.In another embodiment, the VLIW format includes both an ALU operation code that instructs a mathematical function to be performed by the ALU of an execution lane (where, as described below, more than one conventional ALU operation may be specified in one embodiment) and a memory operation code (instructing a memory operation for a specific execution lane or group of execution lanes).

[0031] The term "execution lane" refers to a group of one or more execution units capable of executing an instruction (e.g., logic circuitry capable of executing an instruction). However, in various embodiments, an execution lane may include more processor-like functionality than just execution units. For example, in addition to one or more execution units, an execution lane may also include logic circuitry that decode a received instruction, or, in the case of more MIMD-like architectures, logic circuitry that fetches and decodes an instruction. With respect to MIMD-like approaches, although a centralized program control approach has been largely described herein, a more distributed approach may also be implemented in various alternative embodiments (e.g., including, but not limited to, program code and a program controller within each execution lane of the matrix 305).

[0032] The combination of an execution lane matrix 305, a program controller 309, and a two-dimensional shift register structure 306 provides a highly customizable / configurable hardware platform for a wide range of programmable functions. For example, application software developers may be able to program core systems with a wide range of different functional capabilities and dimensions (e.g., stencil sizes) because the individual execution lanes are capable of performing a wide range of functions and readily accessing input image data near any output matrix location.

[0033] In addition to being used as data storage for image data driven by execution lane matrix 305, random access memories 307 may also maintain one or more lookup tables. In various embodiments, one or more scalar lookup tables may also be instantiated within scalar memory 303.

[0034] A scalar lookup operation involves transmitting the same data value from the same lookup table from the same index to all execution lanes within the execution lane matrix 305. In various embodiments, the VLIW instruction format described above is extended to also include scalar opcode that directs a lookup operation performed by the scalar processor into a scalar lookup table. The index specified for use with the opcode may be an immediate operand or retrieved from another data storage location. Regardless, in one embodiment, a lookup operation to a scalar lookup table within scalar memory essentially involves sending the same data value to all execution lanes within the execution lane matrix 305 during the same clock cycle.Further details on the use and operation of lookup tables are provided below.

[0035] Fig. Figure 3b summarizes the VLIW instruction word embodiments described above. As in Fig. 3b, the VLIW instruction word format includes fields for three separate instructions: 1) a scalar instruction 351 executed by the scalar processor; 2) an ALU instruction 352 dispatched and executed in a SIMD fashion by the respective ALUs within the execution lane matrix; and 3) a store instruction 353 dispatched and executed in a partial SIMD fashion (e.g., if execution lanes along the same row in the execution lane matrix share a same random access memory, then the instruction is actually executed by one execution lane from each of the different rows (the format of the store instruction 353 may include an operand identifying which execution lane from each row the instruction is executed from).

[0036] A field 354 for one or more immediate operands is also included. Which of the instructions 351, 352, 353 use which immediate operand information can be identified in the instruction format. Each of the instructions 351, 352, 353 also includes its own input operands and resulting information (e.g., local registers for ALU operations and a local register and memory address for memory access instructions). In one embodiment, the scalar instruction 351 is executed by the scalar processor before the execution lanes within the execution lane matrix execute any of the other instructions 352, 353. That is, the execution of the VLIW word includes a first cycle in which the scalar instruction 351 is executed, followed by a second cycle in which the other instructions 352, 353 may be executed (note that in various embodiments, the instructions 352 and 353 may be executed in parallel).

[0037] In one embodiment, the scalar instructions executed by the scalar processor include instructions issued to the leaf generator to load / store leaves from / to the memory or 2D shift registers of the data computation unit. Here, the operation of the leaf generator may depend on the operation of the line buffer unit or other variables that prevent a lead-time understanding of the number of cycles the leaf generator will execute to complete an instruction issued by the scalar processor. Accordingly, in one embodiment, any VLIW word whose scalar instruction 351 corresponds to the instruction block generator or otherwise issues an instruction also includes no-op (NOOP) instructions in the other two instruction fields 352, 353.The program code then enters a loop of NOOP instructions for instruction fields 352, 353 until the leaf generator completes its loading from / storage to the data calculation unit. Here, upon issuing an instruction to the leaf generator, the scalar processor can set a bit of a lock register, which the leaf generator resets after the instruction is completed. During the NOOP loop, the scalar processor monitors the lock bit. Once the scalar processor detects that the leaf generator has completed its instruction, normal execution begins again.

[0038] Fig. 4 shows an embodiment of a data calculation component 401. As in Fig. As shown in Figure 4, the data computation component 401 includes a matrix of execution lanes 405 logically positioned "above" a two-dimensional shift register matrix structure 406. As described above, in various embodiments, an image data sheet provided by a sheet generator is loaded into the two-dimensional shift register 406. The execution lanes then operate on the sheet data from the register structure 406.

[0039] The execution lane matrix 405 and the shift register structure 406 are fixed with respect to each other. However, the data within the shift register matrix 406 shifts in a strategic and coordinated manner to cause each execution lane in the execution lane matrix to process a different template within the data. Accordingly, each execution lane determines the output image value for a different pixel in the generated output sheet. From the architecture of Fig. 4, it should be clear that overlapping templates are arranged not only vertically but also horizontally, since the execution lane matrix 405 has vertically adjacent execution lanes as well as horizontally adjacent execution lanes.

[0040] Some architectural features of the data calculation unit 401 to note include the shift register structure 406, which has wider dimensions than the execution lane matrix 405. That is, there is a "ring" of registers 409 outside the execution lane matrix 405. Although the ring 409 is shown on two sides of the execution lane matrix, the ring may exist on fewer (one) or more (three or four) sides of the execution lane matrix 405, depending on the implementation. The ring 405 serves to provide "overflow space" for data that overflows outside the boundaries of the execution lane matrix 405 when the data is shifted "below" the execution lanes 405. In a simple case, a 5×5 stencil centered on the right edge of the execution lane matrix 405 requires four ring register locations further to the right when processing the left-side pixels of the stencil. To simplify the drawing, Fig. 4 the registers of the right side of the ring with only horizontal sliding connections and registers of the bottom of the ring, since only vertical sliding connections are present, when in a nominal embodiment the registers on both sides (right, bottom) would have both horizontal and vertical connections.

[0041] Additional overflow space is provided by random access memories 407 associated with each row and / or column in the matrix, or portions thereof (e.g., a random access memory may be allocated to a "region" of the execution lane matrix spanning 4 lanes row-wise and 2 lanes column-wise. For simplicity, the remainder of the application primarily refers to row- and / or column-based allocation schemes). If the core system operations of an execution lane require processing pixel values ​​outside of the two-dimensional shift register matrix 406 (which some image processing routines may require), the level of image data may further overflow, for example, from the ring region 409 into the random access memory 407. For example, consider a 6×6 template where the hardware includes a ring region of only four memory elements to the right of an execution lane at the right edge of the execution lane matrix.In this case, the data would have to be shifted further to the right from the right edge of ring 409 to completely process the stencil. Data shifted outside of ring area 409 would then overflow into random access memory 407. Other applications of random access memory 407 and the stencil processor are shown in . Fig. 3 are provided below.

[0042] The Fig. 5a to Fig. 5k show an embodiment of the manner in which the image data is shifted within the two-dimensional shift register matrix “below” the execution lane matrix, as indicated above. As in Fig. As shown in Figure 5a, the data contents of the two-dimensional shift matrix are represented in a first matrix 507, while the execution lane matrix is ​​represented by a frame 505. Furthermore, two adjacent execution lanes 510 are shown in simplified form within the execution lane matrix. In this simplified representation 510, each execution lane includes a register R1 that can accept data from the shift register, accept data from an ALU output (e.g., act as an accumulator through cycles), or write output data to an output destination.

[0043] Each execution lane also has the contents "below" the two-dimensional shift matrix in a local register R2. Thus, R1 is a physical register of the execution lane, while R2 is a physical register of the two-dimensional shift register matrix. The execution lane includes an ALU that can operate on operands provided by R1 and / or R2. As described in more detail below, in one embodiment, the shift register is actually implemented with multiple (a "depth") memory / register elements per matrix location, but the shifting activity is limited to one level of memory elements (e.g., only one level of memory elements can shift per cycle). Fig. 5a to Fig. 5k represents one of these deeper register locations, used to store the resulting X from the respective execution lanes. For illustration, the deeper resulting register is drawn next to, rather than below, its counterpart register R2.

[0044] The Fig. 5a to Fig. 5k focus on the computation of two templates whose central position is aligned with the pair of trajectory positions 511 represented in the trajectory matrix. For ease of illustration, the pair of trajectories 510 are drawn as horizontally adjacent, although according to the following example, they are actually vertically adjacent.

[0045] As initially in Fig. As shown in Figure 5a, the execution paths are centered on their central template locations. Fig. Figure 5b shows the object code executed by both execution paths. As in Fig. As shown in Figure 5b, the program code for both execution lanes causes the data to shift one position down and one position to the right within the shift register matrix. This aligns both execution lanes to the upper left corner of their respective templates. The program code then causes the data (in R2) to be loaded into R1 at its respective locations.

[0046] As in Fig. As shown in Figure 5c, the program code next causes the pair of execution lanes to shift the data within the shift register array one unit to the left, causing the value to shift to the right of the respective execution lane position into the respective execution lane position. The value in R1 (previous value) is then added to the new value that has shifted to the execution lane position (in R2). The result is written to R1. As shown in Fig. 5d, the same process as above for Fig. 5c is repeated, causing the result R1 to now contain the values ​​A + B + C in the upper lane and F + G + H in the lower lane. At this point, both lanes have processed the top row of their respective templates. Note the overflow into a ring region on the left side of the lane matrix (if one exists on the left side) or into random access memory if no ring region exists on the left side of the lane matrix.

[0047] As in Fig. As shown in Figure 5e, the program code next causes the data within the shift register array to shift up one unit, causing both execution lanes to align with the right edge of the middle row of their respective templates. Register R1 of the two execution lanes currently contains the sum of the top row of the template and the right value of the middle row. Fig. 5f and Fig. 5g show further progress when moving across the middle row of the two templates of the execution pathway to the left. The cumulative addition continues, so that at the end of the processing of Fig. 5g both execution paths contain the sum of the values ​​of the top row and the middle row of their respective templates.

[0048] Fig. Figure 5h shows a further shift to align each execution lane with the next row of its corresponding templates. Fig. 5i and Fig. 5j show a continuous shift to complete processing across the templates of the two execution pathways. Fig. Figure 5k shows an additional shift to align each execution lane to its correct position in the array and write the result to it.

[0049] In the example from Fig. 5a- Fig. 5k, it should be noted that the object code for the shift operations may include an instruction format that identifies the direction and magnitude of the shift expressed in (X, Y) coordinates. For example, the object code for a one-position shift upward may be expressed in the object code as SHIFT 0, +1. As another example, a one-position shift to the right may be expressed in the object code as SHIFT +1, 0. Furthermore, in various embodiments, larger-size shifts may be specified in the object code (e.g., SHIFT 0, +2). Here, if the 2D shift register hardware only supports one-position shifts per cycle, the machine's instruction may be interpreted to require multi-cycle execution, or the 2D shift register hardware may be designed to support shifts of more than one position per cycle.Embodiments of the latter variant are described in more detail below.

[0050] Fig. Figure 6a shows a further, more detailed representation of the unit cell for the execution lane matrix and the shift register structure (registers in the ring area do not contain a corresponding execution lane). The execution lane and the register area associated with each position in the execution lane matrix are implemented in one embodiment by Fig. 6a is instantiated at each node of the execution path matrix. As in Fig. As shown in Figure 6a, the unit cell includes an execution lane 601 coupled to a register file 602 consisting of four registers R2 through R5. During a cycle, the execution lane 601 can read from or write to any register R1 through R5. For instructions requiring two input operands, the execution lane can fetch both operands from any one of R1 through R5.

[0051] In one embodiment, the two-dimensional shift register structure is implemented by allowing, during a single cycle, the contents of any (only) one of the registers R2 through R4 to be shifted "out" to one of its adjacent register files by the output multiplexer 603, and replacing the contents of any (only) one of the registers R2 through R4 with the contents shifted "in" from a corresponding neighbor by the input multiplexer 604, such that shifts between the neighbors occur in the same direction (e.g., all execution lanes to the left, all execution lanes to the right, etc.). Although it may be common for a same register to have its contents shifted and replaced by the contents shifted in the same cycle, the multiplexer arrangement 603, 604 allows different shift source and shift destination registers within a same register file during a same cycle.

[0052] As in Fig. As shown in Figure 6a, note that during a move sequence, an execution lane moves the contents of its register file 602 to its respective left, right, top, and bottom neighbors. In conjunction with the same move sequence, the execution lane also moves the contents of its register file from a particular one of its left, right, top, and bottom neighbors. The move-out destination and move-in source should again match a common move direction for all execution lanes (e.g., if the move out is to the right neighbor, the move in should be from the left neighbor).

[0053] Although in one embodiment, the contents of only one register per execution lane may be shifted per cycle, other embodiments may allow the contents of more than one register to be shifted in / out. For example, the contents of two registers may be shifted out / in during the same cycle if a second instance of the Fig. 6a observed multiplexer circuit 603, 604 into the design of Fig. 6a. Of course, in embodiments where the contents of only one register can be shifted per cycle, shifts of multiple registers between mathematical operations can occur by consuming more clock cycles for shifts between mathematical operations (e.g., the contents of two registers can be shifted between mathematical operations by consuming two shift operations between the mathematical operations).

[0054] If less than the entire contents of an execution lane's register files are relocated during a relocation sequence, note that the contents of each execution lane's unmoved registers remain in place (are not relocated). Accordingly, any unmoved content that is not replaced by relocated content remains local to the execution lane throughout the relocation cycle. The memory unit ("M") observed in each execution lane is used to load / store data from / to the random access memory associated with the row and / or column of the execution lane within the execution lane matrix. Here, the M unit acts as a standard M unit, frequently being used to load / store data that cannot be loaded / stored from / to the execution lane's own register space.In various embodiments, the primary operation of the M unit is to write data from a local register to memory and to read data from memory and write it to a local register.

[0055] With respect to the ISA opcodes supported by the ALU unit of hardware execution lane 601, in various embodiments, the mathematical opcodes supported by the hardware ALU are integrally related (i.e., substantially equal) to the mathematical opcodes supported by the virtual execution lane (e.g., ADD, SUB, MOV, MUL, MAD, ABS, DIV, SHL, SHR, MIN / MAX, SEL, OR, XOR, NOT). As described above, memory access instructions may be executed by execution lane 601 to retrieve / store data from / to its associated random access memory. In addition, hardware execution lane 601 supports shift operations (right, left, up, down) to move data within the two-dimensional shift register structure. As described above, program control instructions are largely executed by the scalar processor of the stencil processor. c. Compiler-managed stencil processor memory

[0056] With further reference to Fig. 4, it is recalled the existence of a ring area of ​​the registers 409, which in the specific embodiment consists of Fig. 4 around the right and bottom edges of the execution lane matrix 405 (other embodiments may include ring registers on more or fewer sides of the execution lane matrix). In the preceding discussions, a "leaf" was understood to be a matrix of image information that consumed the internal shift register area having the dimensions of the execution lane matrix 405. In various embodiments, the register locations in the ring 409 here do not include associated ALU units for executing image processing instructions on the image data—rather, the ring area 409 serves, for example, to accumulate "overflow" data from shift activity within the execution lane matrix area. Accordingly, the area of ​​image data that can be processed corresponds to area 405, while a "leaf" is thus understood to be coextensive with area 405.Loading a sheet into register matrix 406 involves loading image data into area 405.

[0057] A compiler is responsible for inserting instructions into the program code that essentially control the loading of sheets of information from / to the sheet generator to / from the internal RAM 407 of the data calculation unit 401 of a stencil processor. Accordingly, in various embodiments, the compiler is responsible for determining which sheets from the sheet generator should be loaded into the RAM 407 and when said sheets should be loaded. Additionally, the compiler is responsible for determining which sheet in the stencil processor RAM 407 should be loaded into the two-dimensional shift matrix 406. It should be noted that the RAM 407 can be divided into separate banks, each bank being associated with a specific region of the image with respect to the shift register matrix (e.g.,the RAM banks 407_1 contain image data for the first row of the shift register matrix, the RAM banks 407_2 contain image data for the second row of the shift register matrix, etc.).

[0058] Here, the "reloading" of identical data sheets into RAM 407 from the sheet generator and / or the reloading of identical data sheets from RAM 407 into the two-dimensional shift register 406 should be avoided wherever possible, e.g., to minimize the power consumption of the image processor. As explained in more detail below, inefficiencies can be avoided if the compiler detects the presence of the ring area 409, even though the basic unit of image data is understood as data sheets, which corresponds to the smaller area of ​​the execution lane matrix 405. The avoided inefficiencies are particularly pronounced when the image processing algorithm processes data that extends outside the ring for any output pixels within the execution lane matrix 405.

[0059] The Fig. 7a- Fig. 7d show a basic example. As in Fig. As shown in Figure 7a, a sheet of input image data corresponds to the area 701 of a larger input image. For simplicity, Fig. 7a- Fig. 7d assumes that the input leaf has the same dimensions as the execution lane matrix. Although this arrangement may often be convenient, it is not required. For example, in cases where the resolution of the input leaf differs from that of the output leaf (e.g., due to upsampling or downsampling), the leaves may have different dimensions than the execution lane matrix. For example, in a processor with a 16×16 execution lane matrix, 2× downsampled input leaves may have a dimension of 32×32, while 2× upsampled input leaves may have a dimension of 8×8.

[0060] If the input image is understood to include not only the sheet 701 but also the surrounding image data that could fit within the ring area, the total amount of input image data would consume the shaded area 720. When determining output pixel values ​​for pixels within the sheet 701, at least some algorithms can be designed to process input image data outside the ring area, for example, over the larger area 730.

[0061] For example, if the dimensions of the execution trajectory matrix 405 are 16 pixel locations by 16 pixel locations and the size of the ring region is an additional 4 pixel locations along the X-axis and 4 pixel locations along the Y-axis, leaf 701 would have a dimension of 16 pixel locations by 16 pixel locations and region 720 would have a dimension of 20 pixel locations by 20 pixel locations. If the algorithm that determines the output pixel values ​​for pixels within leaf 701 processes over templates with a dimension of 18 pixels by 18 pixels, the surface area that would need to be processed for leaf 701 would extend outward from region 720 to include the area within boundary 730. For example, to include processing for the pixel location at the lower right corner of sheet 701, a pixel in the lower right corner of the area bounded by boundary 730 would be required.Thus, processing sheet 701 requires significant amounts of data from sheets 702, 703 and 704.

[0062] In one embodiment, the compiler is configured to detect the layout of the multiple leaves within an input image, the dimensions of the execution path matrix, the dimensions of the ring, and the size of the template to be processed for a particular output pixel within the execution path matrix range. The compiler then proceeds to insert memory load instructions at appropriate locations in the code that load a next leaf from the leaf generator into the data computation unit RAM 407.

[0063] In addition, as described in more detail below, the compiler also inserts data movement instructions to move leaves within the random access memory 407 of the data calculation unit to improve their reuse and thereby reduce the duplicate loading of the same leaves into the random access memory 407 from the leaf generator.

[0064] Fig. Figure 7b shows a sequence of instructions 740 constructed by the compiler, which causes the leaf generator to initially load the random access memory 407 with a sufficient number of leaves to fully execute the algorithm for each of the pixel locations of the first leaf location 701. As in Fig. As shown in Figure 7b, the data is loaded from four sheets 701 to 704, after which they are assigned logical sheet identifiers (Lsheet_0, Lsheet_1, LSheet_2, LSheet_3). Here, the number of sheets loaded corresponds to the number of sheets reached by the image processing algorithm when considering the entire shift register matrix 406, including the ring area 409. As can be seen from the discussions of Fig. 3a and Fig. As shown in Figure 3b, the program controller 309, which in various embodiments is associated with the scalar processor 302, executes instructions corresponding to instructions of the sheet generator to load / store data sheets to / from the data calculation unit 301. Accordingly, instructions 740 may be executed by the scalar processor 302.

[0065] The compiler also constructs the subsequent instructions 750, 760 to load the shift register array 406 with the contents from the four sheets just loaded into RAM (MEM LOAD). Instruction 750 fills the shift register area coextensive with the execution lane array 405 with the contents of LSheet_0 (contents of image area 701). Similarly, the scalar processor 302 may also execute instructions 750, 760 in accordance with the above discussions.

[0066] Instruction 760 fills the ring area. That is, instruction 760 performs the following: 1) Load the left side (LHS) of sheet 702 (Lsheet_1) into the ring area 409, which lies directly outside the right edge of the execution lane matrix 405, 2) Load the upper area (UPR) of sheet 704 (Lsheet_3) into the ring area 409, which lies directly below the bottom edge of the execution lane matrix 405, 3) Load the upper left corner of sheet 705 (Lsheet_4) into the ring area from the lower right corner of the execution lane matrix 405. After these loads have been performed, the contents of the entire shift register matrix, including the ring area (area 720 in Fig. 7a), loaded with properly positioned input image data.

[0067] With reference to Fig. 4, it should be noted here that each position of the execution lane matrix contains a memory access unit (M) for executing a memory load instruction (MEM LOAD) to retrieve image data from the random access memory 407. Although in Fig. 4, each of the locations in the ring area 409, in various embodiments, also includes a memory access unit so that image data values ​​can be loaded from the random access memory 407 into the ring area 409. However, again in various embodiments, the ring area locations do not include an ALU unit, so that the actual image processing remains limited to image data within the execution lane matrix 405.Thus, instruction 750 corresponds to a memory load instruction that executes individually at each position of the execution lane matrix, but has different input operand data (which defines the image data to be loaded) based on their position in the matrix, while the MEM LOAD instruction 760 similarly corresponds to a memory load instruction that executes individually at specific positions within the ring area 409, but has different input operand data (which defines the image data to be loaded) based on their position around the execution lane matrix. Techniques for specifying the address for these instructions are described in more detail below.

[0068] The compiler also constructs a subsequent instruction set 770 to perform the image processing algorithm(s) that determine the output pixel values ​​for the pixel locations within the LSheet_0 location. Consistent with the above discussions, the calculations involve operating over a stencil region for each pixel location, accomplished by shifting the contents within the two-dimensional shift register array. The compiler therefore constructs instructions to perform not only the mathematical calculations over the stencil, but also the corresponding register shift instructions.

[0069] In the example currently being discussed, consider that the template to be processed for each pixel location within LSheet_0 has a dimension of 18 × 18. As a result, image data "overflows" from the top and left sides of the execution lane matrix 405 into the random access memory 407. That is, shifts of data to the left result in data from sheet 702 (LSheet 1) and sheet 703 (LSheet_2) within the area bounded by boundary 720 but outside the area bounded by boundary 701 being loaded into the right edge of the ring area 409. Loading the data into the right edge of the ring area 409 effectively shifts "overflow" data from the left edge of the execution lane matrix and the left edge of the ring area 409, which is located directly below the execution lane matrix 405, out of the shift register. The overflowed / shifted out data is written to the random access memory 407.

[0070] Similarly, shifts of data "up" cause data from sheet 704 (LSheet_3) and sheet 703 (LSheet_2) within the area bounded by boundary 720, but outside the execution lane area, to be loaded into the bottom of the circular area 409. Loading data into the bottom of the circular area 409 effectively shifts "overflow" data from the top of the execution lane matrix 409 and the top of the circular area located to the right of the execution lane matrix 405 out of the shift register. The overflowed / shifted-out data is also written to random access memory 407. To effect the shift, however, the compiler constructs the code for loading all new data from RAM 407 into the shift register and writing shifted-out data from the shift register back to RAM 407.

[0071] After all shifts and calculations 770 for sheet 701 (LSheet_0) have been performed, the process continues to scan raster-wise to the right (an output sheet for area 701 can also be written to random access memory 407 for forwarding to the sheet generator). To achieve raster-wise scanning of the algorithm, only a remapping of the logical sheets is required, after which the same program code can be executed again to process output values ​​for the pixels of a next sheet. Accordingly, the compiler's execution of the algorithm by scanning over the image can be constructed as a software loop. Internal pointers in the loop program can be updated with each new iteration of the loop.In particular, when the kernel routine detects that it has reached the very “right” of the data to be loaded for immediate recursion, it also detects that the next recursion is imminent.

[0072] As in Fig. As shown in Figure 7c, the compiler constructs an additional instruction set 780 that prepares the machine to process the algorithm on the next sheet in the raster scan. The instructions include moving (MOV) the contents of LSheet_1 to LSheet_0 and moving the contents of LSheet_2 to LSheet_3. In addition, instructions 780 include instructions to load the next LSheet_1 (sheet 705) and the next LSheet_2 (sheet 706) from the sheet generator into RAM 407. After these instructions 780 have been executed, the machine's context is no different from that of the machine from Fig. 7b prior to the execution of instructions 770, unless the position of the sheet to be processed is that of a sheet to its right (i.e., sheet 702 instead of sheet 701). Importantly, sheets that have already been loaded into RAM 407 (i.e., sheets 702 and 703) are not loaded into RAM 407 from the sheet generator. Accordingly, any inefficiency that would result if sheets 702 and 703 were reloaded into RAM 407 is avoided.

[0073] Note that the loading of LSheet_1 (sheet 705) and LSheet_2 (sheet 706) into RAM 740 from the sheet generator, as indicated by the SG instructions within instructions 780, may be performed, for example, during operations 770 and subsequently moved in RAM to Sheet_1 and Sheet_2 locations after the previous contents of these sheets have been moved to Lsheet_0 and LSheet_3, respectively. Accordingly, the SG LOAD instructions of instructions 780 may actually be implemented as additional MOV instructions within RAM 407. In one embodiment, the move (MOV) instructions of instructions 780 are actual physical data movements within RAM 407. Accordingly, the addresses for LSheet_0 through LSheet_3, as indicated in the program code, are fixed addresses.

[0074] Although the procedure outlined above with respect to the Fig. 7a to Fig. 7c was directed to an example in which the template size to be processed for pixels within leaf 701 was large enough to include pixels located outside the ring region, the method is considered optimal for applications where the template size to be processed for pixels within leaf 701 (i.e., within the execution lane matrix) is small enough that all pixels to be processed lie within the ring region (i.e., pixels outside boundary 720 and within boundary 730 or beyond are not required). In this case, loading additional image data into the two-dimensional shift register (e.g., along the right edge of the ring region) is avoided.

[0075] Fig. 8 shows a method described above. The method includes repeatedly performing 801 the following: Loading a next image sheet from a first location of memory into a two-dimensional shift register array 802. The memory is locally coupled to the two-dimensional shift register array and an execution lane array having a smaller dimension than the two-dimensional shift register array along at least one array axis. The loaded next image sheet remains within an image region of the two-dimensional shift register array. Determining output values ​​for the next image sheet by executing program code instructions along respective lanes of the execution lane array, wherein a template size used in determining the output values ​​includes only pixels that lie within the two-dimensional shift register array 803.Moving a next image data sheet to be fully loaded into the two-dimensional shift register array from a second memory location to the first memory location 804.

[0076] From the above discussion, it is apparent that during the execution of instructions 750, 760, and 770, image data may be loaded from RAM 407 into two-dimensional shift register 406. Here, it is assumed that each individual location in two-dimensional shift register 406 (including the ring area) includes a memory access unit for locally executing a memory load instruction, so that data may be individually loaded from RAM 407 into the local register area. In one embodiment, the compiler fixes a base address component for LSheet_0 in RAM 407, while the compiler-detected addresses of the additional logical sheets for executing the loop (e.g., LSheet_1, LSheet_2, and LSheet_3) have address offsets relative to the base address component.For example, if the address in RAM 407 for LSheet_0 is [base], the address for LSheet_1 is [base] +1, the address for LSheet_2 is [base] +2, and the address for LSheet_3 is [base] +3.

[0077] To support a fully programmable environment, the instruction set architecture of the execution units in the execution lane matrix and in the ring area supports an instruction that generates the correct address offset for the respective shift register matrix location. The address offset can then be used subsequently, for example, to generate the correct address. It is relevant to note here that the example of Fig. 7a to Fig. 7c was simplified in the sense that the leaf dimension was identical to the dimension of the execution path matrix. That is, both the leaves and the execution path matrix had a dimension of 16 pixel locations × 16 pixel locations.

[0078] Other embodiments may, for example, choose to have a sheet size that is larger or smaller than the execution lane matrix along one or both dimensions. In the former case, for example, LSheet_0 extends into the ring area, whereas in the latter case, for example, LSheet_1 and / or LSheet_3 are initially loaded into the dimensions of the execution lane matrix 405. For simplicity, the discussion of the Fig. 9a and Fig. 9b refers to the simplified case where the leaf dimensions are the same as the dimensions of the execution lane matrix. However, as will be made clearer, the generic operation of the instruction can be used for embodiments where the dimensions differ.

[0079] Fig. Figure 9a essentially shows an enlargement of the representation of Fig. 7a, where the full size of the two-dimensional shift register matrix, including its ring area, is shown as a shaded area. Fig. Figure 9b shows the result of a special instruction (QUADRANT) executed at each matrix location. Here, the QUADRANT instruction calculates, for each shift register matrix location, the correct address offset to be added to the base memory address when loading the contents from RAM 407 into the respective shift matrix location.

[0080] In the particular example of Fig. 9b, where the sheet dimensions are the same as the dimensions of the trajectory matrix, all matrix locations associated with the trajectory matrix range will load LSheet_0 with an address offset of 0. In contrast, the matrix locations in the ring range up to the immediate right of the trajectory matrix from LSheet_1 will be loaded with an address offset of +1, the matrix locations in the ring range immediately below the trajectory matrix from LSheet_3 will be loaded with an address offset of +3, while the matrix locations in the ring range outside the edge of the trajectory matrix from LSheet_2 will be loaded with an address offset of +2.

[0081] In one embodiment, the command accepts as input parameters both the X and Y dimensions of the sheet (Xs, Ys) and the X and Y coordinates that specify the position in the matrix (Xa, Ya). Using these values, the QUADRANT command can calculate the address distance as follows: Address distance=0 if (Xa≤Xs) AND (Ya≤Ys)=TRUE1 if (Xa>Xs) AND (Ya≤Ys)=TRUE2 if (Xa>Xs) AND (Ya>Ys)=TRUE3 if (Xa≤Xs) AND (Ya>Ys)=TRUE

[0082] In various embodiments, the compiler generates a matrix of Xa, Ya coordinate pairs that essentially identify the matrix location, reads in the coordinate pair, and specifies Xs and Ys as immediate operands, as these are constant across all executions of the QUADRANT instruction. The Xa and Ya coordinate pairs are loaded into the respective matrix location as input operand information prior to instruction execution. In another embodiment, the instruction additionally accepts the [base] address value as an input operand, so that the full address value [base] + address offset can be provided as the result of the QUADRANT instruction. The [base] operand can be specified as an immediate operand, or the program code can be constructed to dynamically determine the [base] value and pass it as additional input operands to the matrix locations.

[0083] It should be noted that the QUADRANT instruction is specific to an operating environment where the algorithm that determines output pixel values ​​operates on pixel values ​​across four leaves. For operating environments such as those where the algorithm operates across nine leaves, another instruction, HECTANT, can be incorporated into the instruction set architecture, which calculates which of nine different address offsets should be used for any given matrix location.

[0084] Fig. Figure 10 shows an illustration of an alternative embodiment of the raster scan processing approach described above with reference to the Fig. 7a to Fig. 7c. If the sampling in the approach from Fig. 10 to the right, the right side of the ring area is not loaded initially. Instead, only the part of the ring directly below the execution path is loaded. During the operation of the algorithm, as new values ​​from LSheet_1 are needed, for example, to effect a shift to the left, the values ​​are loaded along the right edge of the execution path matrix instead of along the right edge of the ring. The approach from Fig. 10 is particularly useful in cases where the algorithm does not operate outside the ring area and effectively produces code with fewer load instructions. d. Implementation embodiments

[0085] It is important to note that the various features of the image processor architecture described above are not necessarily limited to image processing in the traditional sense and can therefore be applied to other applications, which may or may not require the image processor to be re-characterized. For example, if one of the various features of the image processor architecture described above is to be used in the creation and / or generation and / or playback of animations rather than in the processing of actual camera images, the image processor can be characterized as a graphics processing unit. Furthermore, the architectural features of the image processor described above can be applied in other technical applications, such as video processing, image processing, image recognition and / or machine learning. In this way, the image processor (e.g.as a coprocessor) into a more general purpose processor (e.g. as part of a computer system's CPU) or be a standalone processor within a computer system.

[0086] The hardware embodiments described above may be embodied in a semiconductor chip and / or as a circuit design description for ultimate alignment with a semiconductor manufacturing process. In the latter case, these circuit descriptions may take the form of a (e.g., VHDL or Verilog) description of a register transfer (RTL) circuit, a gating circuit, a transistor circuit, or a mask, or various combinations thereof. Circuit descriptions are typically embodied on a computer-readable storage medium (such as a CD-ROM or other type of storage technology).

[0087] From the previous sections, it can be seen that an image processor, as described above, can be included in the hardware on a computer system (e.g., as part of a handset system on chip (SOC) that processes data from the handset's camera). In cases where the image processor is embodied as a hardware circuit, it should be noted that the image data processed by the image processor can be received directly from a camera. Here, the image processor can be part of a discrete camera or part of a computer system with an integrated camera. In the latter case, the image data can be received directly from the camera or from the computer system's system memory (e.g., the camera sends its image data to the system memory instead of to the image processor). It should also be noted that many of the features described in the previous sections are applicable to an image processing unit (for displaying animations).

[0088] Fig. Figure 11 shows an exemplary representation of a computer system. Many of the components of the computer system described below are applicable to a computer system with an integrated camera and associated image processor (e.g., a handheld device such as a smartphone or tablet computer). Those skilled in the art will easily distinguish between the two.

[0089] As in Fig.11, the basic computer system may include a central processing unit 1101 (which may, for example, include a plurality of general-purpose processing core systems 1115_1 to 1115_N and a main memory controller 1117 arranged on a multi-core processor or an application processor), system memory 1102, a display 1103 (e.g., touchscreen, flat panel display), a locally wired point-to-point connection (e.g., a USB interface) 1104, various network I / O functions 1105 (such as an Ethernet interface and / or a cellular modem subsystem), a wireless local area network (e.g., WLAN) 1106, a wireless point-to-point connection (e.g.,Bluetooth interface) 1107 and a global positioning system interface 1108, various sensors 1109_1 to 1109_N, one or more cameras 1110, a battery 1114, a power management control unit 1112, a speaker and a microphone 1113 and an audio encoder / decoder 1114.

[0090] An application processor or multi-core processor 1150 may include one or more general-purpose processor cores 1115 within its CPUs 1101, one or more graphics processing units 1116, a memory management function 1117 (e.g., a memory controller), an I / O control function 1118, and an image processing unit 1119. The general-purpose processing cores 1115 typically execute the computer system's operating system and application software. The graphics processing units 1116 typically perform graphics-intensive functions, such as generating graphics data to be displayed on the display 1103. The memory control function 1117 is connected to the system memory 1102 to read and write data to and from the system memory 1102. The power management control unit 1112 generally controls the power consumption of the system 1100.

[0091] The image processing unit 1119 may be implemented according to one of the embodiments of the image processing unit described above in the preceding sections. Alternatively, or in combination, the IPU 1119 may be coupled to one or both of the GPU 1116 and the CPU 1101 as a coprocessor thereof. Furthermore, in various embodiments, the GPU 1116 may be implemented with one of the processor features described above.

[0092] The touchscreen display 1103, the communication interfaces 1104-1107, the GPS interface 1108, the sensors 1109, the camera 1110, and the speaker / microphone codec 1113, 1114 can all be considered different forms of I / O (input and / or output) with respect to the overall computing system, including, where appropriate, an integrated peripheral (e.g., the one or more cameras 1110). Depending on the implementation, various of these I / O components may be integrated on the application processor / multi-core processor 1150 or may be located off-chip or outside the package of the application processor / multi-core processor 1150.

[0093] In one embodiment, one or more cameras 1110 include a depth camera capable of measuring the depth between the camera and an object in its field of view. Application software, operating system software, device driver software, and / or firmware executing on a general-purpose CPU core (or other functional block with an instruction execution pipeline for executing program code) of an application processor or other processor can perform all of the functions described above.

[0094] Embodiments of the invention may include various methods, as set forth above. The processes may be embodied in machine-executable instructions. The instructions may be used to cause a general-purpose or special-purpose processor to perform specific processes. Alternatively, these processes may be executed by specific hardware components containing hard-wired logic for executing the processes, or any combination of programmed computer components and custom hardware components.

[0095] Elements of the present invention may further be provided as a machine-readable medium for storing the machine-executable instructions. The machine-readable medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs and magneto-optical disks, FLASH memory, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, propagation media, or other types of media / machine-readable media suitable for storing electronic instructions. For example, the present invention may be downloaded as a computer program that can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communications link (e.g., a modem or network connection) using a data signal embodied in a carrier wave or other propagation medium.

[0096] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the spirit and scope of the invention as set forth in the appended claims. The description and drawings are therefore to be considered in an illustrative rather than a restrictive sense.

Claims

[1] Image processor comprising: a two-dimensional execution path matrix (305; 405); a two-dimensional shift register matrix (306; 406), wherein the two-dimensional execution lane matrix (305; 405) has a smaller dimension than the two-dimensional shift register matrix (306; 406), and wherein the two-dimensional shift register matrix (306; 406) comprises: a region coextensive with the two-dimensional execution path matrix (305; 405); and a ring region (409) of registers peripheral to the two-dimensional execution lane matrix (305; 405); a memory (307; 407) locally coupled to the two-dimensional execution path matrix (305; 405) and the two-dimensional shift register matrix (306; 406), a sheet generator for loading a number of image data sheets into the memory (307; 407), each image data sheet of the number of image data sheets having the same dimensions as the two-dimensional execution path matrix (305; 405); the image processor is configured to execute program code, wherein the program code causes the image processor to perform a method comprising: Loading a first image data sheet (701) of the number of image data sheets from the sheet generator into the memory (307; 407); Loading a second image data sheet of the number of image data sheets from the sheet generator into the memory (307; 407); Loading the first image data sheet (701) from the memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406); Loading a region of the second image data sheet (702) from the memory (307; 407) into the ring region (409), wherein the image content of the second image data sheet (702) is adjacent to the image content of the first image data sheet (701); and Determining output values ​​for the first image data sheet (701) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​includes shifting image data of the first image data sheet (701) and the region of the second image data sheet (702) within the two-dimensional shift register matrix (306; 406), wherein a template size used in determining the output values ​​comprises only pixels that lie within the two-dimensional shift register matrix (306; 406). [2] The image processor according to claim 1, wherein the memory (307; 407) includes a random access memory (307; 407). [3] An image processor according to claim 1 or 2, wherein the method comprises: Writing a first output sheet in the memory (307; 407) for forwarding to the sheet generator, wherein the first output sheet includes the output values ​​for the first image data sheet (701). [4] Image processor according to one of the preceding claims, wherein the first image data sheet has been assigned a first sheet identifier (Lsheet_0) in the memory (307; 407) and the second image data sheet has been assigned a second sheet identifier (Lsheet_1) in the memory (307; 407), and the method comprises: Moving the contents of the second sheet identifier (Lsheet_1) into the first sheet identifier (Lsheet_0); and Loading a third image data sheet (705) of the number of image data sheets from the sheet generator into the memory (307; 407); Loading the second image data sheet (702) from the memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406), Loading a region of the third image data sheet (705) from the memory (307; 407) into the ring region (409), wherein the image content of the third image data sheet (705) is adjacent to the image content of the second image data sheet (702); and Determining output values ​​for the second image data sheet (702) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​includes shifting image data of the second image data sheet (702) and the region of the third image data sheet (705) within the two-dimensional shift register matrix (306; 406), wherein a template size used in determining the output values ​​comprises only pixels that lie within the two-dimensional shift register matrix (306; 406). [5] An image processor according to claim 4, wherein repeated loading of the second image data sheet (702) from the sheet generator is avoided. [6] Image processor according to claim 4 or 5, wherein the loading of the third image data sheet (705) of the number of image data sheets from the sheet generator into the memory (307; 407) is carried out during the determination of output values ​​for the first image data sheet (701) and then the second sheet identifier (Lsheet_1) is assigned to the third image data sheet in the memory (307; 407). [7] An image processor according to any one of claims 4 to 6, wherein the first image data sheet (701) has a base address in the memory (307; 407) and the address of the second image data sheet (702) has an address offset with respect to the base address. [8] An image processor according to any one of claims 4 to 7, wherein the method comprises: Writing a second output sheet in the memory (307; 407) for forwarding to the sheet generator, wherein the second output sheet includes the output values ​​for the second image data sheet (702). [9] Computer system comprising: a) a plurality of general-purpose processor cores (1115_1 to 1115_N); b) a system memory (1102); c) a memory controller (1117) coupled between the system memory (1102) and the plurality of processor cores (1115_1 to 1115_N); d) the image processor according to any one of the preceding claims; and e) a machine-readable storage medium containing the program code. [10] A computer program product for execution by an image processor, the image processor comprising: a two-dimensional execution path matrix (305; 405); a two-dimensional shift register matrix (306; 406), wherein the two-dimensional execution lane matrix (305; 405) has a smaller dimension than the two-dimensional shift register matrix (306; 406), and wherein the two-dimensional shift register matrix (306; 406) comprises: a region coextensive with the two-dimensional execution path matrix (305; 405); and a ring region (409) of registers peripheral to the two-dimensional execution lane matrix (305; 405); a memory (307; 407) locally coupled to the two-dimensional execution path matrix (305; 405) and the two-dimensional shift register matrix (306; 406), a sheet generator for loading a number of image data sheets into the memory (307; 407), each image data sheet of the number of image data sheets having the same dimensions as the two-dimensional execution path matrix (305; 405); wherein the computer program product comprises instructions which, when the program is executed by the image processor, cause the image processor to carry out a method comprising: Loading a first image data sheet (701) of the number of image data sheets from the sheet generator into the memory (307; 407); Loading a second image data sheet of the number of image data sheets from the sheet generator into the memory (307; 407); Loading the first image data sheet (701) from the memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406); Loading a region of the second image data sheet (702) from the memory (307; 407) into the ring region (409), wherein the image content of the second image data sheet (702) is adjacent to the image content of the first image data sheet (701); and Determining output values ​​for the first image data sheet (701) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​includes shifting image data of the first image data sheet (701) and the region of the second image data sheet (702) within the two-dimensional shift register matrix (306; 406), wherein a template size used in determining the output values ​​comprises only pixels that lie within the two-dimensional shift register matrix (306; 406). [11] A computer program product according to claim 10, wherein the method comprises: Writing a first output sheet in the memory (307; 407) for forwarding to the sheet generator, wherein the first output sheet includes the output values ​​for the first image data sheet (701). [12] Computer program product according to claim 10 or 11, wherein the first image data sheet has been assigned a first sheet identifier (Lsheet_0) in the memory (307; 407) and the second image data sheet has been assigned a second sheet identifier (Lsheet_1) in the memory (307; 407), and the method comprises: Moving the contents of the second sheet identifier (Lsheet_1) into the first sheet identifier (Lsheet_0); and Loading a third image data sheet (705) of the number of image data sheets from the sheet generator into the memory (307; 407); Loading the second image data sheet (702) from the memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406), Loading a region of the third image data sheet (705) from the memory (307; 407) into the ring area (409), wherein the image content of the third image data sheet (705) is adjacent to the image content of the second image data sheet (702); and Determining output values ​​for the second image data sheet (702) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​includes shifting image data of the second image data sheet (702) and the region of the third image data sheet (705) within the two-dimensional shift register matrix (306; 406), wherein a template size used in determining the output values ​​comprises only pixels that lie within the two-dimensional shift register matrix (306; 406). [13] The computer program product of claim 12, wherein repeated loading of the second image data sheet (702) from the sheet generator is avoided. [14] Computer program product according to claim 12 or 13, wherein the loading of the third image data sheet (705) of the number of image data sheets from the sheet generator into the memory (307; 407) is carried out during the determination of output values ​​for the first image data sheet (701) and the second sheet identifier (Lsheet_1) is subsequently assigned to the third image data sheet in the memory (307; 407). [15] A computer program product according to any one of claims 12 to 14, wherein the method comprises: Writing a second output sheet in the memory (307; 407) for forwarding to the sheet generator, wherein the second output sheet includes the output values ​​for the second image data sheet (702). [16] A machine-readable storage medium on which the computer program product according to any one of claims 10 to 15 is stored. [17] A method for execution by an image processor, the image processor comprising: a two-dimensional execution path matrix (305; 405); a two-dimensional shift register matrix (306; 406), wherein the two-dimensional execution lane matrix (305; 405) has a smaller dimension than the two-dimensional shift register matrix (306; 406), and wherein the two-dimensional shift register matrix (306; 406) comprises: a region coextensive with the two-dimensional execution path matrix (305; 405); and a ring region (409) of registers peripheral to the two-dimensional execution lane matrix (305; 405); a memory (307; 407) locally coupled to the two-dimensional execution path matrix (305; 405) and the two-dimensional shift register matrix (306; 406), a sheet generator for loading a number of image data sheets into the memory (307; 407), each image data sheet of the number of image data sheets having the same dimensions as the two-dimensional execution path matrix (305; 405); the image processor is configured to execute program code, the program code causing the image processor to perform the method comprising: Loading a first image data sheet (701) of the number of image data sheets from the sheet generator into the memory (307; 407); Loading a second image data sheet of the number of image data sheets from the sheet generator into the memory (307; 407); Loading the first image data sheet (701) from the memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406); Loading a region of the second image data sheet (702) from the memory (307; 407) into the ring area (409), wherein the image content of the second image data sheet (702) is adjacent to the image content of the first image data sheet (701); and Determining output values ​​for the first image data sheet (701) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​includes shifting image data of the first image data sheet (701) and the region of the second image data sheet (702) within the two-dimensional shift register matrix (306; 406), wherein a template size used in determining the output values ​​comprises only pixels that lie within the two-dimensional shift register matrix (306; 406). [18] A method according to claim 17 comprising: Writing a first output sheet in the memory (307; 407) for forwarding to the sheet generator, wherein the first output sheet includes the output values ​​for the first image data sheet (701). [19] Method according to claim 17 or 18, wherein the first image data sheet has been assigned a first sheet identifier (Lsheet_0) in the memory (307; 407) and the second image data sheet has been assigned a second sheet identifier (Lsheet_1) in the memory (307; 407), and the method comprises: Moving the contents of the second sheet identifier (Lsheet_1) into the first sheet identifier (Lsheet_0); and Loading a third image data sheet (705) of the number of image data sheets from the sheet generator into the memory (307; 407); Loading the second image data sheet (702) from the memory (307; 407) into the area of ​​the two-dimensional shift register matrix (306; 406), Loading a region of the third image data sheet (705) from the memory (307; 407) into the ring area (409), wherein the image content of the third image data sheet (705) is adjacent to the image content of the second image data sheet (702); and Determining output values ​​for the second image data sheet (702) by executing program code instructions along respective paths of the execution path matrix (305; 405), wherein determining the output values ​​includes shifting image data of the second image data sheet (702) and the region of the third image data sheet (705) within the two-dimensional shift register matrix (306; 406), wherein a template size used in determining the output values ​​comprises only pixels that lie within the two-dimensional shift register matrix (306; 406). [20] The method of claim 19, wherein repeated loading of the second image data sheet (702) from the sheet generator is avoided. [21] Method according to claim 19 or 20, wherein the loading of the third image data sheet (705) of the number of image data sheets from the sheet generator into the memory (307; 407) is carried out during the determination of output values ​​for the first image data sheet (701) and subsequently the second sheet identifier (Lsheet_1) is assigned to the third image data sheet in the memory (307; 407). [22] Method according to one of claims 12 to 14 comprising: Writing a second output sheet in the memory (307; 407) for forwarding to the sheet generator, wherein the second output sheet includes the output values ​​for the second image data sheet (702).

Images