Image processing apparatus

Abstract

To prevent an increase in calculation cost even if collation areas are large.SOLUTION: An image processing apparatus has: horizontal total sum calculation means (a horizontal total sum addition unit, a horizontal total sum subtraction unit) that calculates the total sum of correlation values for every row in a horizontal direction between a first collation area of a standard image and a first collation area of a reference image, and stores a value for every row based on the total sum in a holding unit; and vertical total sum calculation means (a vertical total sum subtraction unit, a vertical total sum addition unit) that calculates the total sum of correlation values in a vertical direction between the first collation area of the standard image and the first collation area of the reference image, on the basis of the total sum of correlation values for every row in the horizontal direction between the first collation area of the standard image and the first collation area of the reference image. The horizontal total sum calculation means calculates the total sum of correlation values for every row in the horizontal direction between a second collation area of the standard image and a second collation area of the reference image, by using the value for every row stored in the holding unit. The vertical total sum calculation means calculates the total sum of correlation values in the vertical direction between the second collation area of the standard image and the second collation area of the reference image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a processing method for an image processing device, and a program. [Background technology]

[0002] In digital cameras installed as information acquisition sensors for autonomous driving and industrial robots, a technology is known in which pixels with distance measurement capabilities (hereinafter also referred to as "distance measurement pixels") are arranged in some or all of the pixels of the image sensor, and the distance to a subject is detected using a phase difference method.

[0003] In this type of method, multiple photoelectric conversion units are arranged in the ranging pixel, and light beams that pass through different regions on the pupil of the photographing lens are guided to different photoelectric conversion units. Optical images (hereinafter referred to as "Image A" and "Image B", respectively) generated by light beams that pass through different pupil regions can be obtained by the signals output by the photoelectric conversion units included in each ranging pixel, and multiple images can be obtained based on the images A and B. The pupil region corresponding to the image A and the pupil region corresponding to the image B are decentered in different directions along an axis called the pupil division direction.

[0004] Furthermore, a relative positional shift occurs between the multiple acquired images (hereinafter referred to as "A image" and "B image" respectively) along the pupil division direction according to the defocus amount. This positional shift is called image shift, and the amount of image shift is called image shift amount. The distance to the subject can be calculated by converting the image shift amount (parallax) into the defocus amount via a predetermined conversion coefficient. This method, unlike the contrast method, does not require moving the lens to measure the distance, making it possible to measure distances quickly and with high accuracy.

[0005] Disparity calculation generally uses a region-based corresponding point search technique called template matching. In template matching, one of image A or image B is used as the base image, and the other image is used as the reference image. A local region (hereafter referred to as a matching region or block) with a point of interest at its center is set on the base image, and a matching region with a reference point corresponding to the point of interest at its center is also set on the reference image. The reference point is then moved sequentially to search for a point within the matching region where the correlation (i.e. similarity) between image A and image B is highest. Disparity is calculated based on the relative positional shift between this point and the point of interest.

[0006] Various technologies have been developed regarding template matching, and a technology is known that reduces the number of calculations and the number of computing units, and achieves faster processing (see Patent Document 1). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2018-26032 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, while the above-mentioned template matching can reduce the number of calculations by reusing the results of the column-wise area correlation calculations compared to when they are not reused, there is an issue that when the matching area is expanded, the calculation cost increases depending on the size of the matching area.

[0009] An object of the present disclosure is to suppress an increase in computational costs even when the matching area is large. [Means for solving the problem]

[0010] The image processing device has a horizontal summation calculation means for calculating a row-by-row summation of correlation values ​​in the horizontal direction between a first matching area of ​​a standard image and a first matching area of ​​a reference image, and storing a row-by-row value based on the row-by-row summation of correlation values ​​in the horizontal direction in a storage unit, and a vertical summation calculation means for calculating a summation of correlation values ​​in the vertical direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image based on the row-by-row summation of correlation values ​​in the horizontal direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image, and the horizontal summation calculation means uses the row-by-row value stored in the storage unit. the vertical sum calculation means calculates a sum of horizontal correlation values ​​between the second matching area of ​​the standard image and the second matching area of ​​the reference image for each row, and the vertical sum calculation means calculates a sum of vertical correlation values ​​between the second matching area of ​​the standard image and the second matching area of ​​the reference image based on the sum of horizontal correlation values ​​between the second matching area of ​​the standard image and the second matching area of ​​the reference image for each row, wherein the first matching area and the second matching area of ​​the standard image partially overlap each other, and the first matching area and the second matching area of ​​the reference image partially overlap each other. Effect of the Invention

[0011] According to the present disclosure, even if the matching area is large, an increase in computational costs can be suppressed. [Brief description of the drawings]

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0013] (First embodiment) 1 is a diagram showing an example of the configuration of an image processing device 110 according to the first embodiment. The image processing device 110 includes an image input unit 100, a region extraction unit 101, a plurality of correlation value calculation units 104, and a disparity information acquisition unit 105. The region extraction unit 101 includes an edge data extraction unit 102 and a correlation value calculation target region data extraction unit 103.

[0014] The image input unit 100 receives a plurality of image data captured by a stereo camera or the like, and transmits a standard image and a reference image to the area extraction unit 101. The standard image is one of the plurality of image data, and the reference image is the other of the plurality of image data.

[0015] The region extraction unit 101 transmits the base image to the edge data extraction unit 102 , and transmits the reference image to the correlation value calculation target region data extraction unit 103 .

[0016] The edge data extraction unit 102 extracts data on the left end and right end (edge ​​data) of the same line as the pixel of interest in a matching area (hereinafter also referred to as a block) centered on the pixel of interest from the reference image, and transmits the data to the correlation value calculation unit 104.

[0017] To obtain a correlation with the standard image extracted by the edge data extraction unit 102, the correlation value calculation target area data extraction unit 103 extracts data of a target area taking into consideration the search range and block size of the corresponding pixel of interest in the reference image. Specifically, the correlation value calculation target area data extraction unit 103 extracts data of a target area of ​​"search range + horizontal block size × 2-1" and transmits it to the correlation value calculation unit 104. Since image data captured by a stereo camera or the like is handled, the search direction of the reference image is horizontal.

[0018] The multiple correlation value calculation units 104 use the end data of the standard image extracted by the end data extraction unit 102 and the data of the target area of ​​the reference image to calculate a correlation value between the standard image and the reference image for each search position within the search range. A known method can be used to calculate the correlation value. For example, a method called SAD (Sum of Absolute Difference) can be used, in which the sum of the absolute values ​​of the differences between pixel values ​​is used as an evaluation value. In this embodiment, SAD is used, but it is not limited to SAD, and another method such as SSD (Sum of Squared Difference) can be used. A detailed configuration of the correlation value calculation unit 104 will be described later.

[0019] The parallax information acquisition unit 105 receives the correlation values ​​for each search position calculated by the correlation value calculation unit 104, and outputs the image shift amount of the search position with the smallest evaluation value among the correlation values ​​as parallax information.

[0020] Fig. 2 is a diagram showing an example of the configuration of the correlation value calculation unit 104 in Fig. 1. The correlation value calculation unit 104 has a right end absolute difference generation unit 200, a left end absolute difference generation unit 201, a horizontal sum addition unit 202, a horizontal sum subtraction unit 203, and a horizontal correlation value sum holding unit 204. Furthermore, the correlation value calculation unit 104 has a vertical correlation value sum holding unit 205, a vertical sum subtraction unit 206, and a vertical sum addition unit 207.

[0021] The right end absolute difference generation unit 200 generates the absolute difference between the right end data of a block size centered on the pixel of interest of the base image extracted by the end data extraction unit 102 and the right end data of the corresponding reference image for each search position extracted by the correlation value calculation target area data extraction unit 103.

[0022] The left edge absolute difference generation unit 201 generates the absolute difference between the left edge data of a block centered on the pixel of interest of the base image extracted by the edge data extraction unit 102 and the corresponding left edge data of the reference image for each search position extracted by the correlation value calculation target area data extraction unit 103.

[0023] The horizontal sum adder 202 reads out the horizontal correlation value sum excluding the right end absolute difference value from the horizontal correlation value sum hold unit 204, and adds thereto the absolute difference value of the right end data generated by the right end absolute difference generator 200 to calculate a horizontal correlation value sum. The calculated horizontal correlation value sum is transmitted to the horizontal sum subtracter 203 and the vertical correlation value sum hold unit 205.

[0024] The horizontal sum subtraction unit 203 calculates a horizontal correlation value sum intermediate value to be reused in the next column by subtracting the absolute difference value of the left edge data generated by the left edge absolute difference generation unit 201 from the horizontal correlation value sum generated by the horizontal sum addition unit 202. Then, the horizontal sum subtraction unit 203 transmits the horizontal correlation value sum intermediate value to the horizontal correlation value sum holding unit 204.

[0025] The horizontal correlation value sum holding unit 204 is for holding the intermediate value of the horizontal correlation value sum obtained by subtracting the absolute value of the difference at the right end of the block from the sum of the absolute values ​​of the horizontal differences, and transmits the intermediate value to the horizontal sum addition unit 202 and receives the intermediate value from the horizontal sum subtraction unit 203.

[0026] The vertical correlation value sum holding unit 205 stores the horizontal direction correlation value sum generated by the horizontal sum adding unit 202 in a shift register.

[0027] A vertical sum subtraction unit 206 subtracts the horizontal correlation value sum in the front row of the shift register in the vertical correlation value sum holding unit 205 from the correlation value of the entire block, which is the vertical sum calculated by the vertical sum addition unit 207, in order to process the next pixel. The front row of the shift register corresponds to the top pixel position of the block centered on the pixel of interest.

[0028] The vertical sum adder 207 adds the horizontal correlation value sum at the end of the shift register in the vertical correlation value sum holder 205 to the intermediate value of the vertical correlation value sums calculated by the vertical sum subtracter 206, thereby calculating a correlation value for the entire block. The end of the shift register corresponds to the bottom pixel position of the block with the pixel of interest at its center. The correlation value for the entire block is transmitted to the disparity information acquirer 105.

[0029] Specific processing contents will be described below with reference to Fig. 3 to Fig. 5. Fig. 3 is a diagram showing an example of template matching processing by the correlation value calculation section 104 according to this embodiment.

[0030] In this embodiment, the operation will be described using an example where the block size is 5×5 and the search range is 10 (−5 to 4). Fig. 3 shows a standard image B and a reference image C, and since image data captured by a stereo camera is handled, the search direction is horizontal, but the input scanning of the image is performed vertically.

[0031] It is assumed that the pixels of the standard image B and the reference image C are input one pixel at a time from the image input unit 100, with pixel B00 being input, pixel B01 being input in the next cycle, and when the input of one vertical column is completed, pixel B10 in the next column is input.

[0032] If pixel B73 is pixel 301 of interest in reference image B, then in the block on the reference image B side, a 5×5 matching area 302 with pixel B73 at its center is the subject of correlation calculation. In addition, in the block on the reference image C side, 5×5 matching areas with search positions C23 to C113 in search range 303 at their respective centers are the subject of correlation calculation.

[0033] The correlation value calculation unit 104 performs a correlation calculation between the 5 x 5 matching area 302 of the base image B and the 5 x 5 matching area of ​​the reference image C (calculating the sum of the absolute values ​​of the differences between each pixel of the base image B and the reference image C).

[0034] Matching area 304 of reference image C indicates a 5×5 matching area when the search position is -5. Matching area 305 of reference image C indicates a 5×5 matching area when the search position is 4. The smallest result position of the correlation calculation result for each search position becomes the disparity value.

[0035] FIG. 4 is a diagram showing a process of calculating the sum of correlation values ​​in the horizontal direction in the template matching operation of the correlation value calculation unit 104 according to this embodiment.

[0036] If the pixel of interest in standard image B is pixel B73, then edge data extraction unit 102 extracts right edge pixel B93 and left edge pixel B53. In addition, in reference image C, correlation value calculation target region data extraction unit 103 extracts pixel data of correlation calculation region 401, which is pixels C03 to C133.

[0037] Next, the right edge absolute difference generation unit 200 calculates the absolute difference between the extracted right edge pixel B93 and a pixel of the reference image C. The absolute difference calculation is performed for each search position, and when focusing on the matching area 304 at search position “−5”, |B93−C43| becomes the right edge absolute difference 402.

[0038] Similarly, the left edge absolute difference generation unit 201 calculates the absolute difference between the extracted left edge pixel B53 and a pixel of the reference image C. When the matching area 304 at the search position “−5” is focused on, |B53−C03| becomes the left edge absolute difference value 403.

[0039] The horizontal sum addition unit 202 reads out the horizontal correlation value sum intermediate value 405 (|B53 - C03| + |B63 - C13| + |B73 - C23| + |B83 - C33|) excluding the right - end difference absolute value (|B93 - C43|) from the horizontal direction sum holding memory 404 (horizontal correlation value sum holding unit 204), and adds the right - end difference absolute value thereto. By this addition, the horizontal direction correlation value sum (|B53 - C03| + |B63 - C13| + |B73 - C23| + |B83 - C33| + |B93 - C43|) with pixel B73 as the target pixel is completed, and the horizontal direction correlation value sum is transmitted to the vertical correlation value sum holding unit 205.

[0040] The horizontal sum subtraction unit 203 subtracts the left - end difference absolute value (|B53 - C03|) from the above - mentioned horizontal direction correlation value sum calculated by the horizontal sum addition unit 202. Then, the resulting next - column reference horizontal correlation sum value 406 (|B63 - C13| + |B73 - C23| + |B83 - C33| + |B93 - C43|) is stored in the corresponding location (the same line position as the target pixel: mem_addr[3]) of the horizontal correlation value sum holding unit 204.

[0041] The next - column reference horizontal correlation sum value 406, which is the result stored in the horizontal correlation value sum holding unit 204 by the horizontal sum subtraction unit 203, is read out during the scanning of the next column (when the target pixel is B83) and utilized to calculate the horizontal direction correlation value sum of the next column.

[0042] The horizontal correlation value sum holding unit 204 is composed of a memory having addresses corresponding to the vertical size of the input image. Since it is necessary for the search range, when the search range is - 5 to 4, 10 memories are implemented.

[0043] The above - mentioned horizontal direction correlation value sum calculation process is executed in parallel for each search position within the search range. For example, when focusing on the search position "-4", |B93 - C53| becomes the right - end difference absolute value, |B53 - C13| becomes the left - end difference absolute value, and hereinafter, the same process as in the case of the search position "-5" is executed in parallel.

[0044] When the horizontal correlation value sum calculation process for the pixel of interest B73 is completed and the next pixel B74 becomes the pixel of interest, the horizontal correlation value sum for the search range is similarly calculated. For the search position "-5", the horizontal correlation value sum is (|B54-C04|+|B64-C14|+|B74-C24|+|B84-C34|+|B94-C44|).

[0045] After that, the scan continues and the horizontal correlation value sums for the number of search positions within the search range are calculated while reusing the horizontal correlation value sums for the previous row. The calculations required for each pixel to calculate the horizontal correlation value sum are the following four calculations x the number of search ranges:

[0046] Subtraction of absolute difference calculation of the rightmost pixel (operation a) Subtraction of the absolute difference of the leftmost pixel (operation b) Addition of the horizontal sum (operation c) Subtraction of the horizontal correlation sum value calculation based on the next column reference (operation d)

[0047] FIG. 5 is a diagram showing a process of calculating the sum of correlation values ​​in the vertical direction in the template matching operation of the correlation value calculation unit 104 according to this embodiment.

[0048] The vertical correlation value sum is calculated by sequentially storing the horizontal correlation value sum in shift register 503 each time scanning is performed. When the pixel of interest in reference image B is pixel B73, horizontal sum adder 202 calculates the horizontal correlation value sum (|B53-C03|+|B63-C13|+|B73-C23|+|B83-C33|+|B93-C43|) of horizontal correlation value sum calculation region 501 of pixel of interest B73. The calculation result is stored in shift register 503 of vertical correlation value sum hold unit 205.

[0049] When the previous pixel, i.e., the pixel of interest, is pixel B72, the horizontal correlation value sum in horizontal correlation value sum calculation area 502 of pixel of interest B72 calculated by horizontal sum adder 202 is (|B52-C02|+|B62-C12|+|B72-C22|+|B82-C32|+|B92-C42|). This horizontal correlation value sum is stored in the previous register in shift register 503 of vertical correlation value sum hold unit 205.

[0050] 5 shows an example of the configuration of the shift register 503 (vertical correlation value sum holding unit 205), the vertical sum subtraction unit 206, and the vertical sum addition unit 207. The horizontal correlation value sum sad_x output by the horizontal sum addition unit 202 is stored in the shift register 503, and data for the vertical block size is stored.

[0051] The vertical sum addition unit 207 adds the horizontal sum sad_x[4] of the lower end centered on the pixel of interest to the intermediate sum sad_y_tmp to calculate the vertical sum, i.e., the correlation value sad_y of the entire block of the entire block size (5×5). The correlation value sad_y of the entire block needs to be updated. Therefore, the vertical sum subtraction unit 206 calculates the vertical intermediate sum sad_y_tmp of the next pixel by subtracting the horizontal sum sad_x[0] of the upper end centered on the pixel of interest from the correlation value sad_y of the entire block. The calculations required for each pixel to calculate the correlation value sum in the vertical direction are the following two calculations times the search range:

[0052] Addition of the horizontal sum of the lower edge of the pixel of interest: sad_y = sad_y_tmp + sad_x[4] (operation e) Subtract the horizontal sum of the upper edge of the pixel of interest: sad_y_tmp = sad_y - sad_x [0] (operation f)

[0053] FIG. 6 is a diagram showing a comparison of the number of operations per pixel in the template matching operation of the reference technique and this embodiment.

[0054] In the reference technology, when calculating correlation values ​​for a block size of 5 x 5 centered on a pixel of interest, the sum of absolute difference values ​​is first calculated column by column in the vertical direction. Because the calculation is performed column by column, five subtractors are required to calculate the absolute difference values ​​from the standard image and reference image, and four adders are required to calculate the vertical sum of absolute difference values. In addition, four adders are required to store the calculated vertical sum in a shift register and calculate the horizontal sum collectively row by row. In the reference technology, 13 calculators are required for the correlation value calculation calculation per pixel, and if parallel processing for each search position is taken into account, 130 calculators are required.

[0055] In this embodiment, as shown in FIG. 4 and FIG. 5, when calculating correlation values ​​for a block size of 5×5 with a pixel of interest at the center, the calculation is performed on a pixel-by-pixel basis, not on a column-by-pixel basis. Two subtractors are required to calculate the absolute difference value of the end data in the horizontal direction. In addition, four adder-subtractors are required for adding the absolute difference value at the right end in the horizontal direction, subtracting the absolute difference value at the left end in the horizontal direction, adding the horizontal sum value at the bottom end in the vertical direction, and subtracting the horizontal sum value at the top end in the vertical direction. In this embodiment, six calculators are required for the correlation value calculation calculation per pixel, and 60 calculators are required when parallel processing for each search position is taken into consideration. Therefore, the number of calculators required in this embodiment is smaller than the number of calculators required in the reference technology.

[0056] 7A is a diagram showing the difference in the number of operations due to changes in block size in template matching operations of the reference technology and this embodiment. In the reference technology, the number of operations increases as the block size increases. In contrast, in this embodiment, the number of operations does not depend on the block size, so the number of operations remains constant even if the block size is increased.

[0057] 7B is a diagram showing the difference in the number of calculations due to a change in the search range of the reference image in the template matching operation of the reference technique and this embodiment. In both the reference technique and this embodiment, the number of calculations increases in proportion to the increase in the search range.

[0058] The operation flow of this embodiment will be described below. Fig. 8 is a flowchart showing the processing method of the image processing device 110 shown in Fig. 1.

[0059] In step S800, the image input unit 100 inputs the base image and the reference image, and the process proceeds to step S801. At this time, the base image and the reference image are input and scanned in a direction perpendicular to the search direction. In this embodiment, since the search direction is the horizontal direction, the base image and the reference image are input and scanned in the vertical direction.

[0060] In step S801, the edge data extraction unit 102 extracts left edge data and right edge data (edge ​​data) of a matching region centered on a pixel of interest from the reference image, and the process proceeds to step S802.

[0061] In step S802, the correlation value calculation target region data extraction unit 103 extracts, from the reference image, a correlation value calculation region (matching region) for calculating the correlation with the edge data extracted in step S802, and the process proceeds to step S803.

[0062] In step S803, the correlation value calculation unit 104 changes the search position, and the process proceeds to step S804.

[0063] In step S804, the right end absolute difference generating unit 200 calculates the absolute difference between the right end data extracted in step S801 and the corresponding pixel in the correlation value calculation area extracted in step S802, and the process proceeds to step S805.

[0064] In step S805, the left edge absolute difference generating unit 201 calculates the absolute difference between the left edge data extracted in step S801 and the corresponding pixel in the correlation value calculation area extracted in step S802, and the process proceeds to step S806.

[0065] In step S806, the horizontal sum adder 202 reads out the horizontal sum intermediate value of the previous column from the horizontal sum hold memory 404, and the process proceeds to step S807.

[0066] In step S807, the horizontal sum adder 202 calculates the horizontal correlation value sum of the pixel of interest by adding the absolute difference calculated in step S804 to the horizontal sum intermediate value read in step S806, and then proceeds to step S808.

[0067] In step S808, the horizontal sum subtraction unit 203 subtracts the absolute difference value calculated in step S805 from the horizontal correlation value sum calculated in step S807, and stores the result as a horizontal sum intermediate value to be referenced in the next column in the horizontal sum hold memory 404. Then, the process proceeds to step S809.

[0068] In step S809, the horizontal sum adder 202 stores the horizontal direction correlation value sum sad_x calculated in step S807 in the shift register 503 (the vertical correlation value sum holding unit 205), and the process proceeds to step S810.

[0069] In step S810, the vertical sum addition unit 207 adds the horizontal correlation value sum sad_x[4] corresponding to the bottom end of the block centered on the pixel of interest to the intermediate sum sad_y_tmp to calculate the correlation value sad_y of the entire block, and proceeds to step S811.

[0070] In step S811, the vertical sum subtraction unit 206 calculates the intermediate correlation value sad_y_tmp of the entire block to be referenced by the next pixel by subtracting the horizontal correlation value sum sad_x[0] corresponding to the top end of the block centered on the pixel of interest from the correlation value sad_y of the entire block calculated in step S810, and then proceeds to step S812.

[0071] In step S812, the correlation value calculation unit 104 determines whether the correlation value calculation process has been performed at all search positions. If the correlation value calculation process has not been performed at all search positions, the correlation value calculation unit 104 returns to step S803, changes the search position, and repeats steps S804 to S811. If the correlation value calculation process has been performed at all search positions, the correlation value calculation unit 104 proceeds to step S813.

[0072] In step S813, the correlation value calculation unit 104 determines whether correlation value calculation has been performed for all pixels as pixels of interest. If correlation value calculation has not been performed for all pixels as pixels of interest, the correlation value calculation unit 104 returns to step S800, scans the next pixel as the pixel of interest, and repeats the correlation value calculation process. If correlation value calculation processing has been performed for all pixels as pixels of interest, the correlation value calculation unit 104 ends the process of the flowchart in FIG.

[0073] As described above, the image is input and scanned in a direction perpendicular to the search direction, and all correlation value calculations for search positions within the search range are executed in parallel. At this time, the number of calculations can be reduced by reusing the sum of the correlation values ​​of the previous column in the horizontal direction and the sum of the correlation values ​​of the previous row in the vertical direction.

[0074] Furthermore, the memory size for storing the horizontal correlation value sum can be reduced to the vertical size of the input image, making it possible to suppress the circuit scale. If the input is scanned horizontally, the horizontal sum correlation value can be calculated by using a shift register, but the memory required to store the horizontal correlation value sum in order to perform calculations in the vertical direction is (vertical block size - 1) x search range.

[0075] When an SSD is used for the correlation calculation method, the correlation calculation becomes the sum of squared differences, and the number of bits of the sum of correlation values ​​tends to increase. Therefore, it is useful to reduce the amount of memory consumed to hold the sum of correlation values ​​in the horizontal direction by input scanning the image in the vertical direction against the horizontal search.

[0076] Second embodiment The second embodiment will be described below with reference to the drawings. In the first embodiment, an example was shown in which the search range is 10 and the block size is 5×5 to calculate the disparity, but there are also cases in which it is desired to prioritize reducing the number of calculations and suppress memory consumption. A method of thinning out the search range to reduce the number of calculations and suppress memory consumption will be described with reference to Figs. 9 and 10.

[0077] Fig. 9 is a diagram showing a calculation process of a horizontal correlation value sum in the template matching operation of the correlation value calculation unit 104 according to the second embodiment. In the first embodiment, an example in which the search range is 10 was shown, but in the second embodiment, as shown by 901 in Fig. 9, the search range of 10 pixels is thinned out to 5 pixels of {-5, -3, -1, 1, 3}. When performing the horizontal correlation value sum calculation process, the horizontal sum hold memory 902 needs to hold memory for the search range, but by halving the search range, the required capacity of the horizontal sum hold memory 902 is also halved, and only a memory for 5 pixels is required.

[0078] Fig. 10 is a diagram showing a comparison of the number of operations for calculating correlation per pixel in template matching operations of the reference technology and the second embodiment. Fig. 6 shows the number of operations per pixel of the reference technology and the first embodiment when the search range is 10. Fig. 10 compares the number of operations of the reference technology and the second embodiment when the search range is 5, and shows that since the number of operations is proportional to the search range, the number of operations is reduced to half in the second embodiment compared to the number of operations when the search range is 10 shown in the first embodiment.

[0079] As described above, by halving the disparity resolution and reducing the search range, it is possible to reduce the required capacity of the horizontal sum hold memory 902 required for the horizontal correlation value sum calculation process in the template matching operation.

[0080] In this embodiment, the search range is halved from 10 pixels to 5 pixels, but the search range is not limited to this example. Other configurations, operation flows, etc. are the same as those in the first embodiment, so the description will be omitted.

[0081] (Third embodiment) The third embodiment will be described below with reference to the drawings. A method of suppressing memory consumption by thinning out disparity outputs and reducing block sizes will be described with reference to Figs. 11 and 12.

[0082] 11 is a diagram showing a calculation process of the horizontal correlation value sum in the template matching operation of the correlation value calculation unit 104 according to the third embodiment. In the first embodiment, an example in which disparity calculation is performed with a block size of 5×5 has been shown, but in the third embodiment, pixels for which disparity is calculated in the vertical direction are thinned out to a block size of 5×3. A case will be described in which pixel B73 is taken as the pixel of interest and the search range is -5.

[0083] 11, the horizontal direction correlation sum value at the top centered on pixel B73 shown by 1102, and the horizontal direction correlation sum value at the bottom centered on pixel B73 shown by 1103. The correlation value calculation unit 104 calculates the correlation value as the sum of the following 5 × 3 absolute difference values.

[0084] |B51-C01|+|B61-C11|+|B71-C21|+|B81-C31|+|B91-C41|+ |B53-C03|+|B63-C13|+|B73-C23|+|B83-C33|+|B93-C43|+ |B55-C05|+|B65-C15|+|B75-C25|+|B85-C35|+|B95-C45|

[0085] When performing disparity calculation and horizontal correlation value sum calculation processing, the horizontal sum hold memory 1104 needs to have a capacity equivalent to the vertical size of the input image. However, by thinning out the disparity calculation and thinning the block size to 5×3, horizontal correlation sum calculation is performed every two lines, so the required capacity of the horizontal sum hold memory 1104 is halved. 1105 in Fig. 11 indicates a 5×3 matching area at search position -5 corresponding to the pixel of interest B73, and 1106 indicates a 5×3 matching area at search position 4 corresponding to the pixel of interest B73.

[0086] Fig. 12 is a diagram showing a comparison of the number of operations for calculating correlation per pixel in template matching operations between the reference technology and the third embodiment. As previously described in Fig. 7(a), the number of operations in the first embodiment does not depend on the block size, so even if the block size is changed from 5x5 to 5x3 in the third embodiment, the number of operations does not change. In the reference technology, the number of operations depends on the block size, so the number of operations decreases by reducing the block size.

[0087] As described above, by calculating disparity and thinning out the block size, it is possible to reduce the amount of horizontal sum holding memory 1104 required for the horizontal correlation value sum calculation process in the template matching operation.

[0088] In this embodiment, the parallax calculation is thinned in the vertical direction, and the block size is thinned to 5×3, but the size is not limited to this. It is also possible to reduce memory by spatially dividing the image input into upper and lower parts. Other configurations, operation flows, etc. are the same as those of the first and second embodiments, so the description will be omitted.

[0089] As described above, according to the first to third embodiments, the horizontal summation adder 202 and the horizontal summation subtracter 203 are an example of a horizontal summation calculation unit, and the vertical summation subtracter 206 and the vertical summation adder 207 are an example of a vertical summation calculation unit.

[0090] The horizontal sum calculation unit calculates the row-by-row sum of the horizontal correlation values ​​between the first matching area of ​​the standard image and the first matching area of ​​the reference image, and stores row-by-row values ​​based on the row-by-row sums of the horizontal correlation values ​​in the storage unit 204.

[0091] Here, the first matching area in the base image is, for example, a 5×5 matching area 302 centered on pixel B73 of interest. The first matching area in the reference image is, for example, a 5×5 matching area 304 centered on pixel C23.

[0092] The vertical sum calculation unit calculates the sum of the vertical correlation values ​​between the first matching area of ​​the standard image and the first matching area of ​​the reference image based on the row-by-row sum of the horizontal correlation values ​​between the first matching area of ​​the standard image and the first matching area of ​​the reference image.

[0093] Then, the horizontal sum calculation unit uses the row-by-row values ​​stored in the holding unit 204 to calculate the row-by-row sum of the horizontal correlation values ​​between the second matching area of ​​the standard image and the second matching area of ​​the reference image.

[0094] Here, the second matching area of ​​the base image is, for example, a 5×5 matching area centered on pixel B83 of interest. The second matching area of ​​the reference image is, for example, a 5×5 matching area centered on pixel C33. The first matching area and the second matching area of ​​the base image partially overlap each other. The first matching area and the second matching area of ​​the reference image partially overlap each other. Specifically, the second matching area of ​​the base image and the second matching area of ​​the reference image are areas shifted one pixel to the right with respect to the first matching area of ​​the base image and the first matching area of ​​the reference image, respectively.

[0095] The vertical sum calculation unit calculates the sum of the vertical correlation values ​​between the second matching area of ​​the standard image and the second matching area of ​​the reference image based on the row-by-row sum of the horizontal correlation values ​​between the second matching area of ​​the standard image and the second matching area of ​​the reference image.

[0096] The parallax information acquisition unit 105 acquires parallax information based on the sum of the correlation values ​​calculated by the vertical sum calculation unit.

[0097] The horizontal sum subtraction unit 203 stores in the storage unit 204 the row-by-row values ​​obtained by subtracting the leftmost correlation value of each row between the first matching area of ​​the base image and the first matching area of ​​the reference image from the row-by-row sum of the horizontal correlation values ​​between the first matching area of ​​the base image and the first matching area of ​​the reference image.

[0098] The horizontal sum addition unit 202 calculates the sum of the vertical correlation values between the second collation area of the reference image and the second collation area of the reference image by adding the value for each row stored in the holding unit 204 and the correlation value at the right end of each row between the second collation area of the reference image and the second collation area of the reference image.

[0099] The vertical sum subtraction unit 206 calculates an intermediate value sad_y_tmp by subtracting the sum sad_x[0] of the correlation values of the upper end row between the first collation area of the reference image and the first collation area of the reference image from the sum sad_y of the vertical correlation values between the first collation area of the reference image and the first collation area of the reference image.

[0100] The vertical sum addition unit 207 calculates the sum of the vertical correlation values between the third collation area of the reference image and the third collation area of the reference image by adding the intermediate value sad_y_tmp and the sum sad_x[4] of the correlation values of the lower end row between the third collation area of the reference image and the third collation area of the reference image.

[0101] Here, the third collation area of the reference image is, for example, a 5×5 collation area centered on the target pixel B74. The third collation area of the reference image is, for example, a 5×5 collation area centered on the pixel C24. A part of the first collation area and the third collation area of the reference image overlap with each other. A part of the first collation area and the third collation area of the reference image overlap with each other. Specifically, the third collation area of the reference image and the third collation area of the reference image are areas shifted downward by one pixel with respect to the first collation area of the reference image and the first collation area of the reference image, respectively.

[0102] The sum sad_x[0] of the correlation values of the upper end row and the sum sad_x[4] of the correlation values of the lower end row are stored in the shift register 503.

[0103] The above correlation value is, for example, the absolute value of the difference between the pixel value of the reference image and the pixel value of the reference image. Also, the above correlation value may be the sum of the squares of the differences between the pixel value of the reference image and the pixel value of the reference image.

[0104] In the first embodiment, the horizontal summation calculation unit and the vertical summation calculation unit calculate the sum of correlation values ​​while shifting the matching area of ​​the reference image in the horizontal direction (search direction) relative to the first matching area of ​​the standard image.

[0105] In the second embodiment, the horizontal sum calculation section and the vertical sum calculation section calculate the sum of correlation values ​​while thinning and shifting the matching region of the reference image in the horizontal direction relative to the first matching region of the standard image.

[0106] In the third embodiment, the horizontal summation calculation unit calculates a summation of the horizontal correlation values ​​between the first matching area of ​​the base image and the first matching area of ​​the reference image for each thinned row, and calculates a summation of the horizontal correlation values ​​between the second matching area of ​​the base image and the second matching area of ​​the reference image for each thinned row.

[0107] The template matching operation of the correlation value calculation unit 104 has been described above through the first to third embodiments. Since there is a trade-off between the disparity calculation accuracy and the circuit scale (memory usage), a system may be created in which the above-mentioned memory reduction effect and the like are divided into modes and switched depending on the use case.

[0108] For example, if the circuit scale of the target FPGA device is sufficient, disparity calculation is performed without reducing accuracy as in the first embodiment. If the circuit scale of the FPGA device is not sufficient and it is acceptable to reduce the accuracy of disparity calculation, the search range is thinned out to reduce the disparity resolution and reduce the circuit scale as in the second embodiment. If it is not desirable to reduce the disparity resolution, the resolution of the disparity output is reduced to reduce memory usage as in the third embodiment.

[0109] As described above, this is useful when developing a system in which the parallax resolution, resolution, block size, etc. are parameterized and settings are switched according to the priority of the combination.

[0110] The correlation value calculation unit 104 inputs and scans an image in a direction orthogonal to the search range, executes all correlation calculation processes at search positions within the search range in parallel, and can suppress the number of arithmetic operations by reusing the sum of the correlation values in the previous column in the horizontal direction and the sum of the correlation values in the previous row in the vertical direction.

[0111] In addition, since the correlation value calculation unit 104 can manage the memory size for holding the sum of the correlation values in the horizontal direction with only the height of the input image, it is possible to suppress the circuit scale. Also, since the number of arithmetic operations of the correlation value calculation unit 104 does not depend on the block size, it is possible to increase the block size to obtain a noise suppression effect. Further, the correlation value calculation unit 104 is also effective when adjusting the consumption of FPGA resources such as memory according to the implemented design and target device in the case of implementing FPGA implementation.

[0112] The correlation value calculation unit 104 can realize template matching that suppresses an increase in calculation cost even when expanding the collation area.

[0113] (Other embodiments) The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0114] Note that the above-described embodiments are merely specific examples for implementing the present disclosure, and the technical scope of the present disclosure is not limitedly interpreted by these. That is, the present disclosure can be implemented in various forms without departing from its technical idea or its main features.

[0115] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) a horizontal sum calculation means for calculating a sum for each row of a horizontal correlation value between a first matching area of ​​a standard image and a first matching area of ​​a reference image, and storing a value for each row based on the sum for each row of the horizontal correlation value in a storage unit; a vertical sum calculation means for calculating a sum of correlation values ​​in a vertical direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image, said horizontal sum calculation means calculates, by using the values ​​for each row stored in said storage unit, a sum for each row of correlation values ​​in the horizontal direction between the second matching area of ​​said standard image and the second matching area of ​​said reference image; the vertical sum calculation means calculates a sum of correlation values ​​in a vertical direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image; the first matching area and the second matching area of ​​the reference image partially overlap each other; 13. An image processing device, comprising: a first matching area and a second matching area of ​​the reference image, the first matching area and the second matching area being partially overlapped with each other. (Configuration 2) 2. The image processing device according to configuration 1, further comprising: a disparity information acquisition means for acquiring disparity information based on the sum of correlation values ​​calculated by the vertical sum calculation means. (Configuration 3) the horizontal sum calculation means stores in the storage unit a value for each row obtained by subtracting the correlation value at the left end of each row between the first matching area of ​​the standard image and the first matching area of ​​the reference image from a sum for each row of the correlation values ​​in the horizontal direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image; The image processing device according to configuration 1 or 2, characterized in that the horizontal sum calculation means calculates a sum of correlation values ​​in the vertical direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image by adding the value for each row stored in the holding unit and the correlation value at the right end of each row between the second matching area of ​​the standard image and the second matching area of ​​the reference image. (Configuration 4) said vertical sum calculation means calculates an intermediate value obtained by subtracting a sum of correlation values ​​of a top row between the first matching area of ​​said standard image and the first matching area of ​​said reference image from a sum of correlation values ​​in a vertical direction between the first matching area of ​​said standard image and the first matching area of ​​said reference image; said vertical sum calculation means calculates a sum of correlation values ​​in a vertical direction between the third matching area of ​​said standard image and the third matching area of ​​said reference image by adding said intermediate value and a sum of correlation values ​​of a bottom row between the third matching area of ​​said standard image and the third matching area of ​​said reference image; the first matching area and the third matching area of ​​the reference image partially overlap each other; 4. The image processing device according to any one of configurations 1 to 3, wherein the first matching area and the third matching area of ​​the reference image partially overlap each other. (Configuration 5) The image processing device according to configuration 4, characterized in that the sum of the correlation values ​​of the top row between the first matching area of ​​the base image and the first matching area of ​​the reference image and the sum of the correlation values ​​of the bottom row between the third matching area of ​​the base image and the third matching area of ​​the reference image are stored in a shift register. (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the correlation value is an absolute value of a difference between a pixel value of the standard image and a pixel value of the reference image. (Configuration 7) 6. The image processing device according to any one of configurations 1 to 5, wherein the correlation value is a sum of squares of differences between pixel values ​​of the standard image and pixel values ​​of the reference image. (Configuration 8) The image processing device described in configuration 3, characterized in that the second matching area of ​​the base image and the second matching area of ​​the reference image are areas shifted one pixel to the right relative to the first matching area of ​​the base image and the first matching area of ​​the reference image, respectively. (Configuration 9) The image processing device described in configuration 4, wherein the third matching area of ​​the base image and the third matching area of ​​the reference image are areas shifted one pixel downward relative to the first matching area of ​​the base image and the first matching area of ​​the reference image, respectively. (Configuration 10) The image processing device according to any one of configurations 1 to 9, characterized in that the horizontal sum calculation means and the vertical sum calculation means calculate a sum of correlation values ​​while shifting a matching area of ​​the reference image in the horizontal direction relative to a first matching area of ​​the standard image. (Configuration 11) The image processing device according to any one of configurations 1 to 9, characterized in that the horizontal sum calculation means and the vertical sum calculation means calculate the sum of correlation values ​​while thinning out and shifting the matching area of ​​the reference image in the horizontal direction relative to a first matching area of ​​the standard image. (Configuration 12) The image processing device according to any one of configurations 1 to 11, characterized in that the horizontal sum calculation means calculates a sum of thinned rows of horizontal correlation values ​​between a first matching area of ​​the base image and a first matching area of ​​the reference image, and calculates a sum of thinned rows of horizontal correlation values ​​between a second matching area of ​​the base image and a second matching area of ​​the reference image. (Method 1) a horizontal sum calculation step of calculating a sum for each row of horizontal correlation values ​​between a first matching area of ​​a standard image and a first matching area of ​​a reference image, and storing a value for each row based on the sum for each row of the horizontal correlation values ​​in a storage unit; a vertical sum calculation step of calculating a sum of correlation values ​​in a vertical direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image; a second horizontal sum calculation step of calculating a sum for each row of correlation values ​​in a horizontal direction between a second matching area of ​​the standard image and a second matching area of ​​the reference image, using the values ​​for each row stored in the storage unit; a second vertical sum calculation step of calculating a sum of correlation values ​​in a vertical direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image, the first matching area and the second matching area of ​​the reference image partially overlap each other; A processing method for an image processing device, wherein the first matching area and the second matching area of ​​the reference image partially overlap each other. (Program 1) 13. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12. [Explanation of symbols]

[0116] 100 image input unit, 101 area extraction unit, 102 edge data extraction unit, 103 correlation value calculation target area data extraction unit, 104 correlation value calculation unit, 105 parallax information acquisition unit

Claims

1. a horizontal sum calculation means for calculating a sum for each row of horizontal correlation values ​​between a first matching area of ​​a standard image and a first matching area of ​​a reference image, and storing a value for each row based on the sum for each row of the horizontal correlation values ​​in a storage unit; a vertical sum calculation means for calculating a sum of correlation values ​​in a vertical direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image, the horizontal sum calculation means calculates, for each row, a sum of correlation values ​​in a horizontal direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image, using the values ​​for each row stored in the storage unit; the vertical sum calculation means calculates a sum of correlation values ​​in a vertical direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image; the first matching area and the second matching area of ​​the reference image partially overlap each other; 13. An image processing apparatus comprising: a first matching area and a second matching area of ​​the reference image, the first matching area and the second matching area being partially overlapped with each other.

2. 2. The image processing apparatus according to claim 1, further comprising: a parallax information acquisition unit that acquires parallax information based on the sum of the correlation values ​​calculated by the vertical sum calculation unit.

3. the horizontal sum calculation means stores in the storage unit a value for each row obtained by subtracting the correlation value at the left end of each row between the first matching area of ​​the standard image and the first matching area of ​​the reference image from a sum for each row of the horizontal correlation values ​​between the first matching area of ​​the standard image and the first matching area of ​​the reference image; 2. The image processing device according to claim 1, wherein the horizontal sum calculation means calculates a sum of correlation values ​​in the vertical direction between the second matching area of ​​the base image and the second matching area of ​​the reference image by adding together the value for each row stored in the holding unit and the correlation value at the right end of each row between the second matching area of ​​the base image and the second matching area of ​​the reference image.

4. said vertical sum calculation means calculates an intermediate value obtained by subtracting a sum of correlation values ​​of a top row between the first matching area of ​​said standard image and the first matching area of ​​said reference image from a sum of correlation values ​​in a vertical direction between the first matching area of ​​said standard image and the first matching area of ​​said reference image; said vertical sum calculation means calculates a sum of correlation values ​​in a vertical direction between the third matching area of ​​said standard image and the third matching area of ​​said reference image by adding said intermediate value and a sum of correlation values ​​of a bottom row between the third matching area of ​​said standard image and the third matching area of ​​said reference image; the first matching area and the third matching area of ​​the reference image partially overlap each other; 2. The image processing apparatus according to claim 1, wherein the first matching area and the third matching area of ​​the reference image partially overlap each other.

5. 5. The image processing device according to claim 4, wherein the sum of the correlation values ​​of the top row between the first matching area of ​​the base image and the first matching area of ​​the reference image and the sum of the correlation values ​​of the bottom row between the third matching area of ​​the base image and the third matching area of ​​the reference image are stored in a shift register.

6. 2. The image processing apparatus according to claim 1, wherein the correlation value is an absolute value of a difference between a pixel value of the standard image and a pixel value of the reference image.

7. 2. The image processing apparatus according to claim 1, wherein the correlation value is a sum of squares of differences between pixel values ​​of the standard image and pixel values ​​of the reference image.

8. 4. The image processing device according to claim 3, wherein the second matching area of ​​the base image and the second matching area of ​​the reference image are areas shifted one pixel to the right relative to the first matching area of ​​the base image and the first matching area of ​​the reference image, respectively.

9. 5. The image processing device according to claim 4, wherein the third matching area of ​​the base image and the third matching area of ​​the reference image are areas shifted one pixel downward relative to the first matching area of ​​the base image and the first matching area of ​​the reference image, respectively.

10. 2. The image processing device according to claim 1, wherein the horizontal sum calculation means and the vertical sum calculation means calculate the sum of the correlation values ​​while shifting the matching area of ​​the reference image in the horizontal direction with respect to the first matching area of ​​the base image.

11. 2. The image processing device according to claim 1, wherein the horizontal sum calculation means and the vertical sum calculation means calculate the sum of correlation values ​​while thinning out and shifting the matching area of ​​the reference image in the horizontal direction with respect to the first matching area of ​​the base image.

12. 2. The image processing device according to claim 1, wherein the horizontal sum calculation means calculates a sum of each thinned row of horizontal correlation values ​​between a first matching area of ​​the base image and a first matching area of ​​the reference image, and calculates a sum of each thinned row of horizontal correlation values ​​between a second matching area of ​​the base image and a second matching area of ​​the reference image.

13. a horizontal sum calculation step of calculating a sum for each row of horizontal correlation values ​​between a first matching area of ​​a standard image and a first matching area of ​​a reference image, and storing a value for each row based on the sum for each row of the horizontal correlation values ​​in a storage unit; a vertical sum calculation step of calculating a sum of correlation values ​​in a vertical direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the first matching area of ​​the standard image and the first matching area of ​​the reference image; a second horizontal sum calculation step of calculating a sum for each row of correlation values ​​in a horizontal direction between a second matching area of ​​the standard image and a second matching area of ​​the reference image, using the values ​​for each row stored in the storage unit; a second vertical sum calculation step of calculating a sum of correlation values ​​in a vertical direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image based on a row-by-row sum of correlation values ​​in a horizontal direction between the second matching area of ​​the standard image and the second matching area of ​​the reference image, the first matching area and the second matching area of ​​the reference image partially overlap each other; A processing method for an image processing apparatus, wherein the first matching area and the second matching area of ​​the reference image partially overlap each other.

14. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12.

Images