Hardware accelerator for integral image computation

Abstract

A method for computing integral image values of an image in a hardware accelerator includes computing (2400) a row sum value for each row of a row block of the image, wherein the row sum value for each row is computed in parallel in a respective row computation component of the hardware accelerator, computing (2402) an integral image value for the row block using the row sum value for each row and a block pivot from a previous row block of the image, storing (2404) a last row of the integral image value as a block pivot for computing an integral image value for a next row block of the image, and storing (2406) the integral image value for the row block in a memory.

Description

BACKGROUND

[0001] Integral images (also known as summed area tables) are used in computer vision processing to compute sums of pixel values or rectangular subsets thereof in an image. The value at any location (x, y) of an integral image is the sum of the value of the pixel at location (x, y) in the input image and all pixel values to the left and above the pixel in the input image. Once an integral image is computed, finding the sum of pixel values over any rectangular region requires four array references and can be performed at constant speed, regardless of the region size.

[0002] Integral images are particularly useful for fast implementation of image pyramids in multi-scale computer vision algorithms, as such images are the underlying data structure used for feature computation. For example, using integral images improves the execution speed of computing box filters used for feature recognition, as integral images eliminate computationally expensive multiplications and allow all box filters to be computed at constant speed regardless of size.

[0003] While the computation of integral images involves only additions, the total number of operations is large due to the size of the input image. Thus, computing integral images in real-time embedded computer vision systems negatively impacts performance and consumes limited hardware resources. Furthermore, due to the size of the integral image values, which are larger than the original pixel values, using integral images substantially increases memory size and consumes a large amount of memory bandwidth, which is also undesirable in embedded computer vision systems. SUMMARY

[0004] Embodiments of the present description relate to methods and apparatus for accelerating integral image computation. In one aspect, a hardware accelerator for computing integral image values of an image is provided, including a plurality of row computation components configurable to operate in parallel to compute row sum values for respective rows of a block of rows of the image.

[0005] In one aspect, a method for computing integral image values of an image in a hardware accelerator is provided, including: computing row sum values for each row of a block of rows of the image, wherein the row sum values for each row are computed in parallel in respective row computation components of the hardware accelerator; computing integral image values for the block of rows using the row sum values for each row and a block pivot from a previous block of rows of the image; storing a last row of the integral image values as a block pivot for computing integral image values of a next block of rows of the image; and storing the integral image values for the block of rows in a memory.

[0006] In one aspect, a system on a chip (SOC) is provided that includes a hardware accelerator to compute integral image values for a row block of an image, the hardware accelerator configurable to compute row sum values for each row of the row block in parallel; a memory coupled to the hardware accelerator to store the integral image values; and a processor coupled to the memory and the hardware accelerator, the processor operable to execute a control program comprising software instructions that configure the hardware accelerator to generate the integral image values for the row block. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a simple example illustrating integral image computation;

[0008] Figure 2 is a simple example illustrating pixel magnitude computation using integral image values;

[0009] Figure 3 is a high level block diagram of an example multi-processor system on a chip (SOC);

[0010] Figure 4 is a simple example illustrating pivot boundary data used in integral image computation;

[0011] Figures 5 to 18 is an example of block-based integral image computation using pivot boundary data;

[0012] Figure 19 is an example illustrating row sum computation of N values in log2N steps;

[0013] Figure 20 is a block diagram of an example integral image accelerator of the SOC of Figure 3

[0014] Figure 21 is a block diagram of an example integral image accelerator of the SOC of Figure 3

[0015] Figure 22 is a block diagram of an example row sum computation component of the integral image accelerator of Figure 20 or Figure 21

[0016] Figure 23 is a flowchart of a method for computing a row sum value for a row of pixels of an image; and

[0017] Figure 24 is a flowchart of a method for computing an integral image. DETAILED DESCRIPTION

[0018] ​​​Particular embodiments of the present description will now be described in detail with reference to the drawings. Like elements in the various figures are denoted by the same reference numerals for consistency.

[0019] As mentioned previously, the value at any location (x, y) of the integral image is the sum of the values of the pixels in the image at location (x, y) and all the pixel values to the left and above the pixel location (x, y) in the image. Figure 1 is a simple example of the integral image value computation region for a location (x, y), where the "shaded" region covers the image pixel values in the image that are summed to compute the integral image value for location (x, y). The value for an integral image location corresponding to a location in the image can be computed using a recursive dependent serial computation as follows:

[0020] Row_Sum(x, y) = Row_Sum(x, y - 1) + pixel(x, y) (1)

[0021] II(x, y) = II(x - 1, y) + Row_Sum(x, y) (2)

[0022] where II is the integral image, and Row_Sum computes the sum of all pixels in the row of the image before the pixel at location (x, y).

[0023] Given the integral image, the sum of pixel values within a rectangular region of the original image can be computed in constant time with four array references, regardless of the region size, which is useful for computer vision processing such as feature detection. For example, for feature detection, the difference in total pixel magnitude between two image regions can be computed. Referring to Figure 2 A simple example of two example regions, the total pixel magnitude of the two example regions can be computed as follows

[0024] M ABCD = II(A + D - B - C)

[0025] M CDEF = II(C + F - D - E)

[0026] where A, B, C, D, E, and F are the integral image (II) values at the corners of the regions.

[0027] Computing the integral image in an embedded computer vision system faces both computation throughput and memory bandwidth challenges. Embodiments of the present description provide a hardware accelerator for integral image computation that reduces both the computation time and memory bandwidth required to compute the integral image.

[0028] Figure 3is a high-level block diagram of an example multi-processor system on a chip (SOC) 300 configured to support computer vision processing. SOC 300 includes dual general purpose processors (GPPs) 302, dual digital signal processors (DSPs) 304, a vision processor 306, and an integral image (II) accelerator 312 coupled via a high-speed interconnect 322.

[0029] Vision processor 306 is a vector processor tuned for various computer vision and other signal processing tasks, such as speeded up robust features (FAST) feature computation, deep neural convolutional network computation, etc. GPPs 302 host an operating system and provide overall control of the operation of SOC 300. DSPs 304 provide support for computer vision processing, such as object detection and classification, which can use integral images generated using II accelerator 312.

[0030] SOC 300 further includes a direct memory access (DMA) component 308, a camera capture component 310 coupled to an external camera 324, a display management component 316, on-chip random access (RAM) memory 318 (e.g., a computer-readable medium), and various input / output (I / O) peripherals 320, all coupled to processors 302, 304, and II accelerator 312 via interconnect 322. Camera 324 can be coupled to SOC 300 via an analog connection, an Ethernet cable, a low voltage differential signaling (LVDS) cable, or other suitable wired or wireless connection.

[0031] As described in detail in Figures 20 to 22 II accelerator 312 includes functionality to generate integral image values for blocks of rows of images captured by camera 324. A control program executing on DSPs 304 configures II accelerator 312 to generate integral image values for each block of rows of an input image according to parameters specified by a computer vision application in response to requirements of the application. The parameters of II accelerator 312 are described in more detail herein. II accelerator 312 generates integral image values for blocks of rows according to the configured parameters and stores the integral image values in RAM 318 for use by the application.

[0032] The architecture of II accelerator 312 is based on the following observations about the computation of integral image values. One observation is that an image can be divided into smaller blocks for integral image computation as long as certain pivot boundary data is shared between adjacent blocks. Consider a block of an image that is bordered on the left by a block of the same size. The integral image value for the left border of the block is the sum of the pixel values in the left border of the block and the pixel values in the left border of the adjacent block. The integral image value for the right border of the block is the sum of the pixel values in the right border of the block and the pixel values in the right border of the adjacent block. The integral image value for the top border of the block is the sum of the pixel values in the top border of the block and the pixel values in the top border of the adjacent block. The integral image value for the bottom border of the block is the sum of the pixel values in the bottom border of the block and the pixel values in the bottom border of the adjacent block. Figure 4A simple example of the above is shown in FIG. 5. The example image is divided into four row blocks, i.e., four sets of non-overlapping sequence rows, which are each divided into four non-overlapping blocks. The positions corresponding to the pivot boundary data produced during the row-by-row block computation of the integral image are indicated by the "shaded" boxes at the right edge and bottom of the blocks. The pivot boundary data item corresponding to the rightmost column of each block is the Row_Sum value at the respective boundary position (see equation 1 above), and is referred to herein as a row pivot. The pivot boundary data item corresponding to the bottom row of each row block is the actual integral image value of the bottom row, and is referred to herein as a block pivot. The row pivots and block pivots can be used to compute the integral image values for the blocks of the sequence image row.

[0033] Figures 5 to 15 A simple example of the above is shown in FIG. 5. The example image is divided into four row blocks, i.e., four sets of non-overlapping sequence rows, which are each divided into four non-overlapping blocks. The positions corresponding to the pivot boundary data produced during the row-by-row block computation of the integral image are indicated by the "shaded" boxes at the right edge and bottom of the blocks. The pivot boundary data item corresponding to the rightmost column of each block is the Row_Sum value at the respective boundary position (see equation 1 above), and is referred to herein as a row pivot. The pivot boundary data item corresponding to the bottom row of each row block is the actual integral image value of the bottom row, and is referred to herein as a block pivot. The row pivots and block pivots can be used to compute the integral image values for the blocks of the sequence image row. Figure 5 The pixel values of an image having nine rows and twelve columns are shown, which are divided into three row blocks 500, 502, 504 of three rows each. Each row block is divided into three blocks having three rows and four columns each. For ease of explanation, the pixel values are repeated in each block.

[0034] The integral image values corresponding to the top row block 500 are determined as follows. Each row of the row block 500 is divided into three row segments each having four pixels, and the row sum of the pixel values in each row segment is determined. Figure 6 The results of the row sum determinations are shown. Next, the row sum values in each row sum segment are converted to row sum values for the entire respective original row. To accomplish this, the row pivots for each row sum segment are determined. The row pivot for a row sum segment is the sum of the row pivot of any preceding row sum segment and the last row sum value in the row sum segment. For example, in the top row of FIG. 5, the row pivot for the first row sum segment is 10, since there is no preceding row sum segment. The row pivot for the middle row sum segment is 20, which is the sum of the last value in the middle row sum segment and the row pivot of the preceding row sum segment. The row pivot for the third row sum segment is 30, which is the sum of the last value in the third row sum segment and the row pivots of the two preceding row sum segments. The row pivots for the row sum segments of the other two rows can be similarly determined. Figure 6 Figure 7 The row pivots for each of the row sum segments of FIG. 5 are shown. Figure 6

[0035] To convert the row sum values in each row sum segment to row sum values for the entire respective original row, for each row sum segment, the row pivot of the preceding row sum segment is added to each value in the row sum segment. For example, in the top row of FIG. 5, the row sum value for the first row of the entire original row is 10, which is the sum of the row pivot of the first row sum segment and the first value in the first row sum segment. The row sum value for the second row of the entire original row is 20, which is the sum of the row pivot of the first row sum segment and the second value in the first row sum segment. The row sum value for the third row of the entire original row is 30, which is the sum of the row pivot of the first row sum segment and the third value in the first row sum segment. The row sum values for the other two rows of the entire original row can be similarly determined. Figure 6 ​​In the top row, the first row and segment of the top row do not have preceding rows and segments; therefore, the values ​​in said rows and segments are the row sum values ​​of the rows. For the middle rows and segments, the row pivot value 10 corresponding to the first row and segment is added to each value in the middle row and segment to calculate the row sum value of the original row corresponding to said row and segment. For the third row and segment, the row pivot value 20 corresponding to the middle row and segment is added to each value in the third row and segment to calculate the row sum value of the original row corresponding to said row and segment. The row sum values ​​of the corresponding original rows for the row and segment of the other two rows can be calculated similarly. Figure 8 Showing in addition Figure 7 The pivot of the line Figure 6 The sum of rows and segments obtained from the given rows and segments.

[0036] Next, the integral image value corresponding to row block 500 is determined. As shown in equation (2) above, the integral image value of pixel position (x,y) in the image can be determined by adding the row sum at position (x,y) to the integral image value of pixel position (x-1,y). In other words, the integral image value of pixel position (x,y) can be determined by adding the row sum at position (x,y) to all row sums in the same column above position (x,y), i.e., by directly adding the row sums. Therefore, given Figure 8 The row sum and value shown in the figure, the integral image value at any position (x,y) in row block 500 can be determined by adding the row sum and value at position (x,y) to all the row sum and values ​​above it in the same column.

[0037] therefore, Figure 8 The values ​​in the top row are integral image values ​​because there are no rows above the top row. To determine the corresponding... Figure 8 The integral image values ​​of the middle row are used to add the sum of each row in the middle row to the sum of the row in the same column of the top row. This is to determine the values ​​corresponding to... Figure 8 The integral image values ​​of the bottom row are used to add the sum of each row in the bottom row to the sum of the row in the same column of the middle row and the sum of the row in the same column of the top row. Figure 9 Display the integral image value obtained from row block 502. Figure 9 The integral image values ​​in the bottom row 900 are used to determine Figure 5 The next row of block 502 is the block hub for the integral image value. Note that each block hub 900 is the sum of the row sums in the corresponding column of block 500.

[0038] The integral image values ​​of the next row block 502 are determined as follows. Each row of row block 502 is divided into three row segments, each with four pixels, and the row sum of the pixel values ​​in each row segment is determined. Figure 10 The results of the rows and sections are displayed. Next, the row sums in each row and section are converted to the row sums of the entire corresponding original row. To do this, the row pivot for each row and section is determined. Figure 11 exhibitFigure 10 The row hub of each of the rows and segments.

[0039] To convert the row sum value in each row and section into the row sum value of the entire corresponding original row, for each row and section, add the row pivot of the previous row and section to each value in the row and section. Figure 12 Showing in addition Figure 11 The pivot of the line Figure 10 The sum of rows and segments obtained from the given rows and segments.

[0040] Next, the integral image value corresponding to row block 502 is determined. As previously described, the integral image value at any position (x, y) in a row block can be determined by adding the row sum value at (x, y) to all the row sum values ​​above it in the same column of the image, i.e., by directly adding the row sum values. Furthermore, each block pivot from the previous row block 500 is the sum of the row sums in the corresponding column of the previous row block 500. Therefore, given... Figure 12 The integral image value at any position (x, y) in row block 502, as shown in the diagram, can be determined by adding the row sum at position (x, y) to all the row sums above it in the same column of row block 502 and row block 500. Furthermore, since each block hub 900 from row block 500 is the sum of the row sums in the corresponding column of row block 500, the integral image value at any position (x, y) in row block 502 can be determined by adding the row sum at position (x, y) to all the row sums above it in the same column of row block 502 and to the block hub 900 from row block 500 corresponding to the same column.

[0041] Therefore, in order to determine the corresponding Figure 12 The integral image values ​​of the top row are used to add the sum of each row in the top row to the block hub 900 in the corresponding column. This is to determine the value corresponding to... Figure 12 The integral image values ​​of the middle row are added to the row sum values ​​of the same column in the top row and the block hub 900 in the corresponding column. This is to determine the values ​​corresponding to... Figure 12 The integral image values ​​of the bottom row are added to the row sums of each row in the bottom row and the row sums of the same column in the middle row and the same column in the top row, as well as the block hub 900 in the corresponding column. Figure 13 Display the integral image value obtained from row block 502. Figure 13 The integral image values ​​in the bottom row 1300 are used to determine Figure 5 The next row, block 504, is the block hub for the integral image value. Note that each block hub is the sum of the row sums in the corresponding columns of row blocks 500 and 502.

[0042] The integral image values ​​of the next row block 504 are determined as follows. Each row of row block 504 is divided into three row segments, each with four pixels, and the row sum of the pixel values ​​in each row segment is determined. Figure 14The row and segment sums are shown. Next, the row and segment values in each are converted to the row and segment values for the entire corresponding original row. To do this, the row pivots for each row and segment are determined. Figure 15 The row and segment values in each of the row and segment sums are shown. Figure 14 The row pivots for each of the row and segment sums.

[0043] To convert the row and segment values in each row and segment to the row and segment values for the entire corresponding original row, for each row and segment, the row pivot for the previous row and segment is added to each value in the row and segment. Figure 16 The resulting row and segment values for Figure 15 when the row pivots for Figure 14 are added.

[0044] Next, the integral image values corresponding to the row block 504 are determined. As previously described, the integral image value for any position (x, y) in the row block can be determined by adding the row sum value at (x, y) to all the row sum values above it in the same column in the image, i.e., by adding the row sum values straight up. Furthermore, each block pivot 1300 from the previous row block 502 is the sum of the row sums in the corresponding column of row block 500 and row block 502. Thus, given the row and segment values shown in Figure 16 the integral image value for any position (x, y) in the row block 504 can be determined by adding the row sum value at position (x, y) to all the row sum values above it in the same column in row block 504, row block 502, and row block 500. Furthermore, because the block pivots 1300 from row block 502 are each the sum of the row sums in the corresponding column of row block 500 and row block 502, the integral image value for any position (x, y) in the row block 504 can be determined by adding the row sum value at position (x, y) to all the row sum values above it in the same column in row block 504 and the block pivots 1300 from row block 502 corresponding to the same column.

[0045] Thus, to determine the integral image value corresponding to the top row of Figure 16 each row sum value in the top row is added to the block pivot 1300 in the corresponding column. To determine the integral image value corresponding to the middle row of Figure 16 each row sum value in the middle row is added to the row sum values in the same column in the top row and the block pivot 1300 in the corresponding column. To determine the integral image value corresponding to the bottom row of Figure 16 each row sum value in the bottom row is added to the row sum values in the same column in the middle row and the same column in the top row and the block pivot 1300 in the corresponding column. Figure 17 The resulting integral image values for row block 504 are shown, and Figure 18 the final integral image for the example image is shown.

[0046] Another observation is that the sum of a sequence of n pixel values can be computed in log2N steps of parallel additions, where if n is a power of 2, then N = n, otherwise N is the nearest power of 2 that is larger than n. For example, if n = 25, then N = 32 or if n = 16, then N = 16. Consider the example of Figure 19 where n = 16 and N = 16, and the values are 1 through 16. The initial step adds the original value pairs with a stride of 1. Thus, each value is added to the immediately preceding value in the set. The arrows illustrate the values that contribute to each sum. In the second step, the results of the stride = 1 additions are added in pairs with a stride of 2. In the third step, the results of the stride = 2 additions are added in pairs with a stride of 4. In the fourth and final step, the results of the stride = 4 additions are added in pairs with a stride of 8. Note that each value resulting from the stride = 8 addition is the row sum for the corresponding pixel position in the original set of values. Note further that each stride after the initial stride is a power of 2 of the previous stride.

[0047] Figure 20 is an example II accelerator 312 that can be used as Figure 3 A block diagram of an example II accelerator 2000 of the II accelerator 312 that can be used as

[0048] The row computation components 2002-2006 are configured to operate in parallel to compute row sum values for a row block of an image. Each row computation component 2002-2006 can be configured to compute a row sum value for a row of a row block. The number of row computation components 2002-2006 and the maximum input row size of each row computation component depends on the implementation and can be determined, for example, based on criteria such as desired computation speed, power consumption, and chip silicon area budget. For example, in some embodiments, there are sixteen row computation components, each of which can accept a maximum input row size of 4096 pixels. Further, in some embodiments, the input row size of each row computation component 2002-2006 can be configurable and can be smaller than the maximum input row size. In some embodiments, the number of row computation components 2002-2006 used to compute the integral image for a row block can also be configurable and can be smaller than the total number of row computation components. Figure 22 The operation of the row computation components 2002-2006 is described in

[0049] The block integral image computation component 2008 includes functionality to receive the row sum values for a row block from the row computation components 2002-2006 and to compute the integral image values for the row block using the row sum values and the block pivots from the previous row block, if any. Reference is made earlier herein to Figures 5 to 18Examples describe a block pivot. Block pivot buffer 2010 stores block pivots from a previous row of blocks. To compute integral image values, block integral image computation component 2008 includes functionality to perform straight-line additions in each column of a row and value of a row block using the block pivot for the corresponding column, as previously described herein with reference to Figures 5 to 18 Examples describe. Note that the number of straight-line additions in each column depends on the number of rows in the input row block. In some embodiments, the straight-line additions are implemented as a network of adders.

[0050] Block integral image computation component 2008 is further configured to output integral image values of the input row block (which are stored in external memory (e.g. Figure 3 RAM 318) and store the bottom row of integral image values in block pivot buffer 2010 for use as block pivots for the next input row block.

[0051] Figure 21 is an example II accelerator 2100 that can be used as Figure 3 a II accelerator 312 of FIG. 3. Example II accelerator 2100 includes functionality to compute integral image values for a plurality of consecutive rows (i.e., a row block) in an image. II accelerator 2100 includes a plurality of row computation components 2102-2106 and a block pivot buffer 2110. Multiple invocations of II accelerator 2100 can be performed to produce an integral image for an entire image.

[0052] Row computation components 2102-2106 are configured to operate in parallel to compute row sums and values for a row block of an image. Each row computation component 2102-2106 can be configured to compute a row sum and value for a row of a row block. The number of row computation components 2102-2106 and the maximum input row size of each row computation component depends on the implementation and can be determined, for example, based on criteria such as desired computation speed, power consumption, and chip silicon area budget. For example, in some embodiments, there are sixteen row computation components, each of which can accept a maximum input row size of 4096 pixels. Further, in some embodiments, the input row size of each row computation component 2102-2106 can be configurable and can be less than the maximum input row size. In some embodiments, the number of row computation components 2102-2106 used to compute an integral image for a row block can also be configurable and can be less than the total number of row computation components. Figure 22 The operation of row computation components 2102-2006 is described in

[0053] DSP 304( Figure 3 ) is coupled to II accelerator 2100 to receive row sums and values for a row block from row computation components 2102-2106 and includes functionality to compute integral image values for the row block using the row sums and values and block pivots from a previous row block (if any). As previously described herein with reference to Figures 5 to 18Examples describe a block pivot. To compute the integral image values, the DSP 304 includes functionality to perform straight-line additions in each column of the row sum and values of the row block using the block pivot of the corresponding column, as previously described Figures 5 to 18 in examples. Note that the number of straight-line additions in each column depends on the number of rows in the input row block.

[0054] The DSP 304 is further configured to output the integral image values of the input row block (which are stored in external memory (e.g. Figure 3 RAM 318) and store the bottom row of integral image values in memory accessible by the DSP 304 for use as the block pivot for the next input row block.

[0055] Figure 22 is a block diagram of an example row computation component 2200, such as Figure 20 the row computation components 2002-2006 in the II accelerator 2000 of Figure 21 the row computation components 2102-2106 in the II accelerator 2100 of FIG. 1. The row computation component 2200 includes functionality to compute the row sum and values of a sequence of pixels (e.g., a row of pixels from an image). The example row computation component 2200 includes a row data router 2202, a plurality of pixel processing components 2204-2210, a plurality of segment row sum and computation components 2214-2220, a row pivot computation component 2222, a row sum computation component 2224, and a configuration component 2212.

[0056] The input data router 2202 includes functionality to receive a sequence of pixel values from an image (e.g., a row of pixel values) and route a configurable number of segments of sequence pixel values to respective pixel processing components 2204-2210. Each pixel processing component 2204-2210 includes functionality to receive a segment of sequence pixel values from the input data router 2202 and perform any processing needed to prepare the pixel values for computation by the respective segment row sum and computation component 2214-2220. For example, in some embodiments, if the input pixel values are 8-bit or 12-bit values, the pixel processing components 2204-2210 align the values to be 16-bit.

[0057] Each segment row sum and compute component 2214-2220 includes functionality to compute the row sum and value for a segment of pixel values received from the corresponding pixel processing component 2204-2210. The output of each segment row sum and compute component 2214-2220 is a segment of row sum and values corresponding to the input segment of pixel values. For example, assume there are three segment row sum and compute components SRSC1, SRSC2, and SRSC3. Further, assume the input segment for SRSC1 is [1 2 3 4], the input segment for SRSC2 is [5 6 7 8], and the input segment for SRSC3 is [9 10 11 12]. The output segment for SRSC1 is [1 36 10], the output segment for SRSC2 is [5 11 18 26], and the output segment for SRSC3 is [9 19 30 42]. See also Figures 5 to 18 Examples in this regard are described with respect to determining row sums for segments.

[0058] In some embodiments, the segment row sum and compute components 2214-2220 include functionality to perform segment row sum computation in log2N steps of parallel addition, as described with respect to the example of Figure 19 In such embodiments, N is the number of pixel values in the input segment received from the corresponding pixel processing component 2204-2210.

[0059] The number of pixel processing components and segment row sum and compute components included in a row compute component depends on the implementation. Further, the maximum number of pixel values that can be processed by these components also depends on the implementation. In some embodiments, the maximum expected row width of an input image and implementation complexity are factors considered in determining the number of pixel processing components and segment row sum and compute components and the maximum number of pixel values that each component can process. In some embodiments, the expected maximum row width is 4096 pixels, and the row compute component includes sixteen pixel processing components and sixteen segment row sum and components, each of which is configured to accept a maximum of 256 pixels as input.

[0060] The row pivot compute component 2222 includes functionality to compute the row pivots needed to compute the row sum for the entire input row. To compute the row pivots, the row pivot compute component 2222 receives the last or rightmost row sum and value from each enabled row sum compute component 2214-2220. The row pivot compute component 2222 uses the received row sum and values to compute the row pivot for each segment of row sum values. The row pivots can be computed recursively as row_pivot(N) = pivot_data(N) + row_pivot(N-1), where N is the Nth row segment and pivot_data is the last or rightmost row sum and value of the Nth row segment. Continuing the example above, the pivot values from the row segments are 10, 26, and 42, and the computed row pivot values are 10, 36, and 68. See also Figures 5 to 18Examples in this regard are determining row pivots.

[0061] It is noted that calculating row pivots using pivot values is analogous to row sum calculation for each row segment. In some embodiments, row pivot calculation component 2222 includes functionality to perform row pivot calculation in log2N steps of parallel addition, as described with reference to Figure 19 Examples, where N is the number of row segments if the number of row segments is a power of 2, otherwise N is the nearest power of 2 greater than the number of row segments.

[0062] Row sum calculation component 2224 includes functionality to calculate the row sum value for an input row using the row sums and segments from segment row sum calculation components 2214-2220 and the row pivots from row pivot calculation component 2222. More specifically, row sum calculation component 2224 includes functionality to add the row pivot to the row sum value in the corresponding row sum segment to yield the row sum value for the entire input row. Continuing the previous example, row integral image calculation component 2126 adds row pivot 10 to row sum segment [5 11 18 26] and adds row pivot 36 to row sum segment [9 19 30 42]. The resulting row sum for the entire row is [1 3 6 10 15 21 28 36 45 55 66 78]. See also Figures 5 to 18 Examples in this regard are determining row pivots.

[0063] Configuration component 2212 receives and stores configuration parameters for row calculation component 2200. Configuration parameters can include, for example, the number of pixels in a row of an input image, the number of bits in a pixel, and the number of pixels in each row segment.

[0064] Figure 23 is a flowchart of a method for calculating a row sum value for a row of pixel values of an input image, such as can be performed by row calculation component 2200. First, a row of pixel values is divided 2300 into segments of sequential pixel values, such as by row data router 2202. Next, row sums for the pixel values in each row segment are calculated 2302 in parallel, such as in respective segment row sum calculation components 2214-2220. In some embodiments, segment row sum calculation is performed in log2N steps of parallel addition, as described herein.

[0065] Next, row pivots are calculated 2304, such as in row pivot calculation component 2222. As described previously herein, the row pivot for a segment is determined using the last or rightmost row sum value for each segment. In some embodiments, row pivot calculation is performed in log2N steps of parallel addition, as described herein.

[0066] Next, the row sums of the segments are used, e.g., in row and compute component 2224, to compute 2306 the row sum values for the rows. As previously described, the row sum values for a row are computed by adding the row pivot to the row sum values of the respective row segments.

[0067] Figure 24 is a flowchart of a method for computing an integral image that can be performed, e.g., using II accelerator 312. First, the row sum values for each row of a block of rows of the image are computed 2400 in parallel, e.g., in respective row compute components 2102 to 2106. Next, the integral image values for the block of rows are computed 2402 using the row sum values for each row and the computed block pivot from the previous block of rows. Note that for the initial block of rows of the image, there will be no block pivot. To compute the integral image values, as previously described herein, a column-wise addition is performed in each column of row sum values of the block of rows using the block pivot for the corresponding column. The computation of the integral image values can be performed, e.g., by block integral image compute component 2008 or by DSP 304.

[0068] The last row of integral image values is stored 2404 as the block pivot for the next block of rows, and the integral image values for the block of rows are also stored 2406, e.g., in RAM 318. If the block integral image compute component 2008 performs the computation of the integral image values, the block pivot can be stored in block pivot buffer 2010, or if the DSP 304 performs the computation of the integral image values, the block pivot can be stored in memory accessible by the DSP 304. Next, the method is repeated 2408 for the next block of rows, if any.

[0069] Other Embodiments

[0070] While this description has provided a limited number of examples, those of skill in the art having the benefit of this description will appreciate that other examples can be devised that do not depart from the scope of the description described herein.

[0071] For example, embodiments of II accelerators are described herein with multiple row compute components that compute row sums for respective rows of a block of rows in parallel. In some embodiments, an II accelerator can include only a single row compute component. In such embodiments, rather than computing integral image values for a block of rows, the integral image values are computed one row at a time, and the integral image value for a row is used as the block pivot for the next row in the image.

[0072] In another example, embodiments of SoCs with a single II accelerator are described herein. In some embodiments, a SoC can include two or more II accelerators, and the II accelerators can be used to compute integral image values for respective blocks of rows of an image or to simultaneously compute integral image values for separate images, e.g., separate levels of an image pyramid or images from separate cameras.

[0073] Accordingly, the appended claims will cover any such modifications of the examples falling within the true scope of the description.

Claims

1. A hardware accelerator for computing integral image values of an image, the hardware accelerator comprising: a plurality of row computation components configurable to operate in parallel to compute row sum values for respective rows of a row block of the image; wherein each of the plurality of row computation components comprises: a plurality of segment row sum computation components coupled to receive respective row segments of sequential pixel values of the respective row of the row block, wherein the plurality of segment row sum computation components are configured to compute row sum values for respective segments in parallel; a row hub computation component coupled to the plurality of segment row sum computation components to receive a last row sum value for each respective segment, wherein the row hub computation component is configured to compute a row hub corresponding to each respective segment using the last row sum value; and a row sum computation component coupled to the plurality of segment row sum computation components to receive the row sum values for each respective segment and coupled to the row hub computation component to receive the row hub corresponding to each respective segment, wherein the row sum computation component is configured to compute the row sum for the respective row by adding the row sum values for each respective segment with the row hub corresponding to a previous respective segment in the respective row of the row block.

2. The hardware accelerator of claim 1, further comprising a block integral image computation component coupled to the plurality of row computation components to receive the row sum values for the respective rows and configured to compute an integral image value for the row block using the row sum values and a block hub from a previous row block of the image.

3. The hardware accelerator of claim 2, wherein the block integral image computation component is further configured to compute an integral image value by adding a row sum value corresponding to the integral image value, all row sum values above corresponding row sum values in a same column of the row block, and a block hub corresponding to the same column.

4. The hardware accelerator of claim 1, wherein a processor coupled to the hardware accelerator to receive the row sum values is configured to compute an integral image value for the row block using the row sum values and a block hub from a previous row block of the image.

5. The hardware accelerator of claim 4, wherein the processor is further configured to compute an integral image value by adding a row sum value corresponding to the integral image value, all row sum values above corresponding row sum values in a same column of the row block, and a block hub corresponding to the same column.

6. The hardware accelerator of claim 1, wherein each segment row sum computation component of the plurality of segment row sum computation components is configured to compute the row sum values for the respective segment in log2N steps of parallel additions, wherein if a number of pixel values is a power of two, N is equal to the number of pixel values in the respective segment, otherwise N is the nearest power of two that is greater than the number of pixel values.

7. The hardware accelerator of claim 1, wherein the row pivot calculation component is further configured to calculate the row pivot in log2N steps of parallel addition, where N equals a number of segments if the number of segments is a power of two, otherwise N is a nearest power of two that is greater than the number of segments.

8. A method for calculating integral image values of an image in a hardware accelerator, the method comprising: calculating a row sum value for each row of a row block of the image, wherein the row sum value for each row is calculated in parallel in a respective row calculation component of the hardware accelerator; calculating an integral image value for the row block using the row sum value for each row and a block pivot from a previous row block of the image; storing a last row of the integral image value as a block pivot for calculating integral image values of a next row block of the image; and storing the integral image value for the row block in a memory; wherein calculating a row sum value further comprises: receiving respective row pixel values of the row block in a respective row calculation component; splitting the respective row pixel values into row segments of sequential pixel values in the respective row calculation component; calculating a row sum value for each of the segments, wherein the row sum value for each of the segments is calculated in parallel in a respective segment row sum calculation component of the respective row calculation component; calculating a row pivot for each of the segments using a last row sum value for each of the segments by a row pivot calculation component included in the respective row calculation component; and calculating a row sum value for the row by a row sum calculation component included in the respective row calculation component by adding the row sum value for each of the segments to a previous row pivot corresponding to a previous segment of the respective row pixel values of the row block.

9. The method of claim 8, wherein calculating an integral image value further comprises calculating the integral image value by a block integral image calculation component included in the hardware accelerator.

10. The method of claim 8, wherein calculating an integral image value further comprises calculating the integral image value by a processor coupled to the hardware accelerator to receive the row sum value for each row.

11. The method of claim 8, wherein calculating an integral image value further comprises calculating an integral image value by adding a row sum value corresponding to the integral image value, all row sum values above a corresponding row sum value in a same column of the row block, and a block pivot corresponding to the same column.

12. The method of claim 8, wherein calculating a row sum value for each segment further comprises calculating the row sum value for each segment in log2N steps of parallel addition, where N equals the number of pixel values in a segment if the number of pixel values is a power of two, otherwise N is a nearest power of two that is greater than the number of pixel values. ​ 13. The method of claim 8, wherein computing row pivots further comprises computing the row pivot for each section in log 2N steps of parallel addition, where N equals the number of sections if the number of sections is a power of 2, otherwise N is the nearest power of 2 that is greater than the number of sections.

14. A system on a chip (SOC) comprising: a hardware accelerator to compute integral image values for a row block of an image, the hardware accelerator configurable to compute, in parallel, row sum values for each row of the row block; a memory coupled to the hardware accelerator to store the integral image values; and a processor coupled to the memory and the hardware accelerator, the processor operable to execute a control program comprising software instructions that configure the hardware accelerator to generate the integral image values for the row block; wherein the hardware accelerator is further configurable to: divide each row of the row block into respective sections of sequential pixel values; compute, in parallel, row sum values for the respective sections; use a last row sum value for a respective section to compute a row pivot corresponding to the respective section; and compute row sum values for each row by adding the row sum values for the respective sections to the row pivots corresponding to a previous respective section.

15. The system on a chip (SOC) of claim 14, wherein the hardware accelerator is further configurable to compute the integral image values for the row block using the row sum values for each row and a block pivot from a previous row block of the image.

16. The system on a chip (SOC) of claim 15, wherein the hardware accelerator is further configurable to compute an integral image value by adding a row sum value corresponding to the integral image value, all row sum values above a corresponding row sum value in a same column of the row block, and a block pivot corresponding to the same column.

17. The system on a chip (SOC) of claim 14, wherein the processor is coupled to the hardware accelerator to receive the row sum values for each row, and is further operable to compute the integral image values for the row block using the row sum values and a block pivot from a previous row block of the image.

18. The system on a chip (SOC) of claim 17, wherein the processor is further operable to compute an integral image value by adding a row sum value corresponding to the integral image value, all row sum values above a corresponding row sum value in a same column of the row block, and a block pivot corresponding to the same column.

19. The system on a chip (SOC) of claim 14, wherein the hardware accelerator is further configurable to compute row sum values for the respective sections in log 2N steps of parallel addition, where N equals the number of pixel values in a respective section if the number of pixel values is a power of 2, otherwise N is the nearest power of 2 that is greater than the number of pixel values. ​ 20. The system on a chip (SOC) of claim 14, wherein the hardware accelerator is further configurable to compute a row pivot corresponding to the respective bin in log 2N steps of parallel addition, where N equals a number of bins in a row if a number of pixel values is a power of two, otherwise N is a nearest power of two that is greater than the number of bins.

Images