Image processing apparatus, image processing method, and storage medium

Abstract

To improve processing of distance measurement.SOLUTION: The image processing apparatus includes an image acquisition unit configured to acquire a first image and a second image having parallax, a correlation acquisition unit configured to acquire a plurality of pieces of correlation information between an image of a standard area in the first image and images of a plurality of reference areas corresponding to the standard area in the second image, a parallax amount calculation unit configured to calculate a parallax amount between the image of the standard area and the images of the reference areas based on the plurality of pieces of correlation information, and a storage unit configured to store pixel data constituting the first image and the second image in a memory for a predetermined period. The correlation acquisition means sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means, connects the pixel data to a plurality of past pixel data each time the pixel data is acquired, and stores the connected pixel data for each address of the memory.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Techniques for detecting the distance to a subject using digital cameras are known for acquiring information in autonomous driving, industrial robots, head-mounted displays, and the like. For example, one method involves placing two digital cameras at a predetermined horizontal distance and comparing the images captured by each camera (hereinafter referred to as the "left image" and "right image"). A positional shift occurs between the left and right images captured in this way, depending on the distance to the subject, and the distance to the subject can be calculated by calculating this positional shift. A region-based corresponding point search technique called template matching is used to calculate the positional shift. In template matching, one of the left or right image is used as a base image, and the other image is used as a reference image. A reference region centered on a point of interest is set on the base image, and a reference region centered on a reference point corresponding to the point of interest is also set on the reference image. The reference point is then sequentially moved to search for the point where the correlation between the reference region in the left image and the reference region in the right image is highest. The amount of positional shift, i.e., the parallax, determined to be highly correlated can be converted into a distance value using a predetermined formula using the distance between the two cameras.

[0003] Template matching and other image processing may be implemented as an LSI. In this case, it is necessary to devise a way to store pixel data so that the image position required for template matching can be referenced while acquiring an image from a camera. For example, in Patent Document 1, data for the reference area on the reference image side and the required search range on the reference image side are stored in a register circuit and then searched. In Patent Document 2, when the parallax calculation position is updated, the data already stored in the register circuit is shifted horizontally and vertically accordingly. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-070055 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-108044 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a problem in that storing pixel data that may be searched for using template matching in a register circuit increases the circuit size. Another method is to prepare a dedicated memory unit such as SRAM, store several lines of pixel data, and then read out the pixel data in the required search range as needed. SRAM stands for Static Random Access Memory. However, this can increase the number of accesses required to read the required data, depending on the configuration, resulting in reduced throughput. For this reason, there has traditionally been room for improvement in distance measurement processing.

[0006] The present invention has been made to solve the above-mentioned problems, and has an object to improve the processing of distance measurement. [Means for solving the problem]

[0007] An image processing device according to one embodiment of the present invention comprises an image acquisition means for acquiring a first image and a second image having parallax, a correlation acquisition means for acquiring a plurality of correlation information between an image of a reference area in the first image and images of a plurality of reference areas in the second image corresponding to the reference area, a parallax calculation means for calculating the amount of parallax between the image of the reference area and the image of the reference area based on the plurality of correlation information, and a storage means for retaining pixel data constituting the first image and the second image in a memory for a predetermined period of time, wherein the correlation acquisition means sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means, concatenates the pixel data with previous pixel data each time it is acquired, and stores the concatenated pixel data for each address in the memory. [Effects of the Invention]

[0008] According to the present invention, the processing of distance measurements can be improved. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of a distance measuring device according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an example of the operation of a correlation calculation unit. [Figure 3] FIG. 10 is a diagram showing the order in which pixel data is received by the correlation calculation unit. [Figure 4] FIG. 10 is a diagram illustrating a line buffer required for template matching. [Figure 5] FIG. 1 is a diagram illustrating a conventional method for reading a line buffer. [Figure 6] FIG. 3 is a diagram showing pixel data stored in a line buffer according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing pixel data read out from a line buffer. [Figure 8] FIG. 10 is a diagram showing a state in which template matching is performed on a partial search area. [Figure 9] FIG. 4 is a diagram showing a control flow for writing to a line buffer according to the first embodiment of the present invention. [Figure 10] FIG. 4 is a diagram showing a read control flow to a line buffer according to the first embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing the overall configuration of a distance measuring device according to a second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing a read control flow to a line buffer according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following describes embodiments of the present invention with reference to the drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0011] <Embodiment 1> FIG. 1 is a diagram showing the overall configuration of a distance measuring device according to a first embodiment of the present invention. The distance measuring device 100 has at least two imaging units, imaging unit 101a and imaging unit 101b, for triangulating the distance to a subject for each pixel of a captured image. The distance measuring device 100 is an example of an image processing device. Hereinafter, the image captured and acquired by imaging unit 101a will be referred to as the left image as a reference image, and the image captured and acquired by imaging unit 101b will be referred to as the right image as a displaced image. The left image and the right image are examples of a first image and a second image having parallax. The imaging units 101a and 101b are examples of image acquisition means for acquiring the first image and the second image having parallax.

[0012] Distance measuring device 100 has a distance image generation unit 102. Distance image generation unit 102 generates a distance map indicating the distance to each pixel based on images from imaging units 101a and 101b. Distance image generation unit 102 is typically composed of an LSI, a CPU, a program executed by the CPU, a memory device, various IOs, etc. LSI is an abbreviation for Large Scale Integration. CPU is an abbreviation for Central Processing Unit. IO is an abbreviation for Input / Output.

[0013] The imaging unit 101a has a lens 103a, an imaging element 104a, and an image transmission unit 105a. The lens 103a is a device that forms an image of a subject on the imaging element 104a at the subsequent stage. The imaging element 104a is an image sensor configured from a CMOS (complementary metal oxide semiconductor) or a CCD (charge coupled device).

[0014] The subject image formed on the image sensor 104a via the lens 103a is converted into an electrical signal by the image sensor 104a. The image transmission unit 105a transmits the electrical signal acquired by the image sensor 104a to the distance image generation unit 102 as pixel data.

[0015] The imaging unit 101b has a lens 103b, an imaging element 104b, and an image transmission unit 105b. Since the lens 103b is similar to the lens 103a, the imaging element 104b is similar to the imaging element 104a, and the image transmission unit 105b is similar to the image transmission unit 105a, detailed descriptions of the lens 103b, the imaging element 104b, and the image transmission unit 105b will be omitted.

[0016] The distance image generating unit 102 includes an image receiving unit 106. The image receiving unit 106 receives pixel data transmitted from the imaging units 101a and 101b.

[0017] The distance image generation unit 102 includes an image correction unit 107. The image correction unit 107 performs preprocessing required for generating a distance map on the pixel data sent from the image reception unit 106. Typical examples of this preprocessing include shading correction, which corrects brightness variations due to peripheral light loss caused by the lenses 103a and 103b, and filtering, which emphasizes correlation.

[0018] The distance image generation unit 102 includes a correlation calculation unit 108. The correlation calculation unit 108 calculates multiple correlations between the left and right images at each pixel position within a certain search range using template matching or the like, as will be described later. The correlation calculation unit 108 is an example of a correlation acquisition means that acquires multiple pieces of correlation information between an image of a reference area in the first image and images of multiple reference areas in the second image that correspond to the reference area.

[0019] The distance image generation unit 102 includes a parallax amount calculation unit 109. The parallax amount calculation unit 109 calculates the amount of parallax at each pixel position by selecting the highest correlation from the multiple correlations calculated by the correlation calculation unit 108. If necessary, the parallax amount calculation unit 109 also performs interpolation below the search resolution using parabolic fitting or the like. The parallax amount calculation unit 109 is an example of a parallax amount calculation means that calculates the amount of parallax between the image in the standard area and the image in the reference area based on the multiple pieces of correlation information.

[0020] The distance image generation unit 102 includes a distance calculation unit 110. The distance calculation unit 110 calculates the distance to the subject at each pixel position using the amount of parallax calculated by the parallax calculation unit 109 and the distance between the two lenses 103a and 103b. The distance image generation unit 102 performs the above distance calculations over the entire image, thereby generating a distance map for the entire image.

[0021] The distance image generating unit 102 also includes an overall control unit 111 that controls each unit, and a storage unit 112 that stores the operation setting values ​​of each unit and buffers intermediate data as needed. The storage unit 112 is an example of a storage means that stores pixel data constituting the first image and the second image in memory for a predetermined period of time.

[0022] FIG. 2 is a diagram showing an example of the operation of the correlation calculation unit 108. FIG. 2(A) is a diagram showing a left image. 200a is a reference position for which distance calculation is to be performed in the left image. FIG. 2(B) is a diagram showing a right image. 200b is a corresponding reference position on the right image that indicates the same coordinates as the reference position 200a in the left image. 201a is an object captured on the left image. 201b is an object captured on the right image that indicates the same thing as the object 201a in the left image. The objects 201a and 201b are located at different coordinate positions in the left and right images due to the arrangement interval between the two lenses 103a and 103b and the distance to the objects.

[0023] Here, two lenses 103a and 103b are arranged with a horizontal gap between them, and subjects 201a and 201b are captured with a horizontal parallax. Distance image generator 102 performs, for example, template matching to calculate the amount of parallax.

[0024] Reference numeral 202 denotes a reference area for performing template matching. Reference area 202 is set to a predetermined size around reference position 200a. Reference numeral 203 denotes a search area for searching for a location similar to reference area 202 in the right image. Because lenses 103a and 103b are arranged with a gap between them in the horizontal direction, the amount of misalignment is searched for horizontally, and search area 203 is an area that is long in the horizontal direction.

[0025] FIG. 2(C) is a diagram showing the reference position in search area 203, where the search position is 0. FIG. 2(D) is a diagram showing the reference position in search area 203, where the search position is 1. FIG. 2(E) is a diagram showing the reference position in search area 203, where the search position is 2. FIG. 2(F) is a diagram showing the reference position in search area 203, where the search position is 3. FIG. 2(G) is a diagram showing the reference position in search area 203, where the search position is 4.

[0026] The following description will be given with reference to Figures 2(C), 2(D), 2(E), 2(F), and 2(G). Reference numeral 204 denotes a reference position on the right image side where template matching is performed. Reference numeral 205 denotes a reference region consisting of pixels surrounding reference position 204. In the example of Figure 2, as shown in Figures 2(C), 2(D), 2(E), 2(F), and 2(G), searches are performed in five reference regions 205 within search region 203, with search positions being 0, 1, 2, 3, and 4.

[0027] Here, correlation is calculated at each position of the five reference regions 205. For example, a known method such as SSD, which calculates the dissimilarity by the sum of squares of the differences between pixel values ​​in the standard region 202 and the reference region 205, is used. SSD is an abbreviation for Sum of Squared Difference. In methods using SSD, the calculated value is the dissimilarity, and the location where the dissimilarity is smallest is the location where the correlation is highest. In the example of FIG. 2, the dissimilarity is 10 in the reference region 205 at search position 3. Since the search position selected as the most similar and having the highest correlation by the disparity amount calculation unit 109 at the subsequent stage is 3, the disparity amount is calculated to be 3. Then, using the information that the disparity amount is 3, the distance to the subject is calculated by the distance calculation unit 110.

[0028] FIG. 3 shows the order of pixel data received by the correlation calculation unit 108. Reference numeral 300 denotes the reception order, typically received line by line starting from the top left of the image. The pixel data received by the correlation calculation unit 108 moves from left to right in the image, and when it reaches the right edge, it moves to the left edge of the line immediately below. FIG. 4 shows the line buffers required for template matching. Template matching requires pixel data in the vicinity of the reference position 200a for which distance calculation is to be performed. Therefore, distance calculation for a certain reference position 200a can only be performed when the reception order 300 has progressed a little further; for example, calculation is performed when pixels 400a and 400b are received. Furthermore, because template matching requires reference to surrounding pixel data, line buffers are provided according to the vertical widths of the reference area 202 and search area 203.

[0029] 401a in FIG. 4 is a line buffer for the left image, and 401b in FIG. 4 is a line buffer for the right image. When compared with the configuration in FIG. 1, the line buffers are configured as part of the storage unit 112. The line buffers are configured using SRAMs or the like, and the line buffer addresses correspond to the horizontal positions of pixels. Furthermore, for the vertical positions of pixels, multiple SRAMs are configured to enable parallel access. For example, if the vertical width of the reference area 202 and the search area 203 is 7, seven SRAMs are prepared for parallel reading, and one SRAM is prepared for writing, accessed independently of the reading, for a total of eight SRAMs. As the reception order 300 progresses vertically, the multiple line buffers provided are used by shifting the seven reads and one write by one line, and cycling the line buffer numbers of the assigned line buffers.

[0030] 5 is a diagram showing how the line buffer is read when performing conventional template matching. The aforementioned line buffers 401a and 401b read pixel data from one address in one cycle, but the pixel data required for template matching at the reference position 200a covers a wide range. Therefore, data from each address is read from the line buffer every cycle and stored while being shifted into a register unit 500 included in a calculation unit (not shown).

[0031] The register unit 500 indicates pixel data to be secured in the calculation unit, and corresponds to the reference area 202 in the left image and the search area 203 in the right image. In order to explain the search area 203, which is the larger of the reference area 202 and the search area 203, Fig. 5 illustrates the line buffer 401b, which is the line buffer of the search area 203.

[0032] Referring to FIG. 5, the operation when the receiving order 300 shown in FIG. 3 advances by one will be described below. 501 is a data discarding operation, which discards unnecessary pixel data for one vertical column. 502 is a data shifting operation, which shifts the data position in the register unit 500 by one pixel. 503 is a line buffer read operation, which reads one pixel of data from one address in the line buffer 401b and stores it in the register unit 500. The line buffer read operation 503 reads one vertical column of data in parallel from each of the line buffers 401b arranged in parallel. 504 is the line buffer read position. Note that one of the line buffers 401b is used for writing, so no read access is performed. Using the pixel data secured in the register unit 500 in this way, the calculation unit performs template matching for each search position shown in FIG. 2.

[0033] 5, however, the register unit 500 must have logic circuits corresponding to the search area 203, which increases the circuit size. Therefore, in this embodiment, the necessary pixel data is collected at the stage of storing the pixel data in the line buffer, so that multiple pixel data can be read in a single read.

[0034] FIG. 6 is a diagram showing pixel data stored in a line buffer according to this embodiment. 600 is a line buffer, and the illustration shows one of the line buffers 401a and 401b described above. 601 is an address, and 602 is the concatenated pixel data stored at address 601. d(x) indicates pixel data at horizontal coordinate x. In this embodiment, the data width that can be stored at each address is expanded, and multiple pixel data are concatenated and stored at each address. That is, the position of address x includes, for example, eight pixel data from horizontal coordinate (x-7) to horizontal coordinate x. Note that "concatenated" here means that the next data is placed without any gap between the previous data and the next data.

[0035] When pixel data is written to the line buffer 600, the data received in the order of reception 300 is written while being concatenated with the previous pixel data. In addition, the previous seven pixel data are held in the shift register for future concatenation. Negative horizontal coordinates can be filled with a fixed value such as 0, or replaced with the value of horizontal coordinate 0. In Figure 6, negative horizontal coordinates are replaced with d(0), which is the data for horizontal coordinate 0. This storage method requires eight times the capacity of the SRAM that makes up the line buffer as before, but if the SRAM is implemented in a predetermined unit size, such as in an FPGA, the implementation cost does not necessarily have to be eight times as high.

[0036] 7 is a diagram showing an example of pixel data read from the line buffer during template matching. As described above, 401a is the line buffer for the left image, and 401b is the line buffer for the right image.

[0037] In the left image line buffer 401a, when data at a certain horizontal address is read, connected pixel data for five horizontal pixels can be extracted. Also, by reading data in parallel from each of the line buffers arranged vertically in parallel, pixel data for the reference area 202 can be prepared in one cycle.

[0038] In the right-image line buffer 401b, when data at a certain horizontal address is read, for example, connected pixel data for eight horizontal pixels can be extracted in one cycle. Here, 700 is defined as a partial search area. The partial search area 700 is a pixel data set that can be prepared in one cycle and is a part of the search area 203. It is assumed that the search area 203 is divided into partial search areas 700 when template matching does not necessarily need to be performed in one cycle due to the input / output throughput of the correlation calculation unit 108. Note that the partial search area 700 may be dynamically changed depending on the number of search ranges for template matching, the unit size constraints of the SRAM area mentioned above, the computational load per processing, the degree of parallelism, etc.

[0039] FIG. 8 is a diagram showing template matching performed on a partial search area 700. FIG. 8(A) is a diagram showing a reference area 202 for performing template matching. FIG. 8(B) is a diagram showing a reference position 205 in the partial search area 700, where the search position is 0. FIG. 8(C) is a diagram showing a reference position 205 in the partial search area 700, where the search position is 1. FIG. 8(D) is a diagram showing a reference position 205 in the partial search area 700, where the search position is 2. FIG. 8(E) is a diagram showing a reference position 205 in the partial search area 700, where the search position is 3. In this example, the horizontal size of the reference area 202 and the reference area 205 is 5, and the partial search area 700 includes pixel data for 8 horizontal pixels. Therefore, correlation is calculated using SSD or the like at each of the four positions of the reference area 205 shown in FIGS. 8(B) to 8(E).

[0040] 9 is a diagram showing write control to the line buffer in the control flow instructed by the integrated control unit 111 according to the first embodiment. The processing in FIG. 9 corresponds to part of the processing in the correlation calculation unit 108 and processing related to writing data to the storage unit 112. This flow is applied to both the operations for the left image and the right image.

[0041] In step S900, the correlation calculation unit 108 initializes the line buffer numbers. When eight vertical line buffers are provided as described above, the correlation calculation unit 108 assigns indexes of numbers 0 to 7 to each of the eight line buffers, and starts using the line buffer with line buffer number 0.

[0042] The processing from step S901 to step S910 is a loop for the y coordinate, and processing is performed for each pixel position according to the reception order 300, and the processing is repeated until it is completed for all y coordinates.

[0043] In step S902, the correlation calculation unit 108 initializes a shift register area that stores data for creating concatenated pixel data. Note that, in exceptional processing (not shown), the shift register area may be initialized using the first pixel data after receiving the first pixel data.

[0044] The processing from step S903 to step S908 is a loop for the x coordinate, and processing is performed for each pixel position according to the reception order 300, and the processing is repeated until it is completed for all x coordinates.

[0045] In step S904, the correlation calculation unit 108 receives data d(x) of the current horizontal coordinate x. In step S905, the correlation calculation unit 108 concatenates the data stored in the shift register with the data received in step S904. In step S906, the correlation calculation unit 108 writes the concatenated data in step S905 to address x of the current line buffer number. In step S907, the correlation calculation unit 108 discards old data d(xn) from the shift register and adds d(x). n is, for example, 4 if five pixels are concatenated for the left image, and 7 if eight pixels are concatenated for the right image. Step S908 indicates the end of the loop corresponding to step S903. That is, the correlation calculation unit 108 sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means in the receiving order 300, concatenates each acquired pixel data with the previous multiple pixel data, and stores the concatenated pixel data for each address in the memory.

[0046] In step S909, the correlation calculation unit 108, upon receiving the information that one horizontal line has been completed in step S908, updates the line buffer number to be written to from the next time. Step S910 indicates the end of the loop corresponding to step S901. This completes the processing for one frame.

[0047] 10 is a diagram showing read control to the line buffer in the control flow instructed by the integrated control unit 111 according to the first embodiment. The processing in FIG. 9 corresponds to part of the processing in the correlation calculation unit 108 and processing related to reading data into the storage unit 112.

[0048] In step S1000, the correlation calculation unit 108 initializes the line buffer numbers. When eight line buffers are prepared for vertical use as described above, the correlation calculation unit 108 assigns indexes of numbers 0 to 7 to each of the eight line buffers and starts using line buffers with line buffer numbers 0 to 6. Since multiple lead line buffers are used in parallel in the vertical direction, seven are allocated.

[0049] The processing from step S1001 to step S1011 is a loop related to the y coordinate, and here processing is performed in order focusing on reference position 200a. This is separate from the position of pixel data received in reception order 300, and it is determined, for example, that the position four lines before the position where pixel data is received is processed as shown in Figure 4. The processing from step S1002 to step S1009 is a loop related to the x coordinate, and is intended to perform processing at each position in the same way as step S1001.

[0050] In step S1003, the correlation calculation unit 108 calculates a read address a for the left image. Because there is a certain deviation between the horizontal coordinate of the reference position 200a and the address to be read in the line buffer, this deviation is corrected in the processing of step S1003. For example, if the horizontal width of the reference area 202 is 5, the correlation calculation unit 108 needs to read the address two positions to the right of the reference position 200a.

[0051] In step S1004, the correlation calculation unit 108 reads the connected data for the left image from the address a calculated in step S1003. At this time, the correlation calculation unit 108 reads from all of the multiple allocated line buffers in parallel, thereby collectively acquiring the pixel data required for the reference area 202.

[0052] In step S1005, the correlation calculation unit 108 calculates a read address b for the right image. The processing in step S1005 corrects the deviation from the reference position 200a in the same way as when the read address a was calculated, and also includes a calculation for changing the position of the partial search area 700, which will be described later.

[0053] In step S1006, the correlation calculation unit 108 reads the connected data for the right image from address b calculated in step S1005. In the process of step S1006, similar to the process of step S1004, the correlation calculation unit 108 reads all of the allocated line buffers in parallel to acquire all of the pixel data required for the partial search area 700. That is, in steps S1004 and S1006, the correlation calculation unit 108 reads the connected pixel data contained in one address of the memory.

[0054] In step S1007, the correlation calculation unit 108 performs correlation calculations such as SSD using the reference area 202 and the partial search area 700. As shown in Fig. 8, the correlation calculation unit 108 performs correlation calculations at multiple locations within the partial search area 700 where this calculation is possible. That is, in step S1007, the correlation calculation unit 108 calculates one or more correlations using connected pixel data.

[0055] In step S1008, the correlation calculation unit 108 determines whether the partial search area 700 has finished scanning the search area 203. If the correlation calculation unit 108 determines that scanning has not finished, the process returns to step S1005, and the read address b is changed so that correlation calculation can be performed on a new partial search area 700. If the correlation calculation unit 108 determines that scanning has finished, the process proceeds to step S1009. Step S1009 indicates the end of the loop corresponding to step S1002. That is, the correlation calculation unit 108 has an adjustment means for adjusting the address read from the memory, and performs multiple reads based on the determination in step S1008, thereby calculating multiple correlations.

[0056] In step S1010, in response to the completion of one horizontal line in step S1009, the correlation calculation unit 108 updates the line buffer number to be read from next time. For example, if the correlation calculation unit 108 used line buffer numbers 0 to 6 last time, it will next use line buffer numbers 1 to 7, and then cyclically use line buffer numbers 2 to 7 and 0 after that. Step S1011 marks the end of the loop corresponding to step S1001. This completes the processing for one frame.

[0057] As described above, in this embodiment, the distance measuring device 100 that calculates distance by performing template matching can perform the required disparity search and distance measurement while reducing the register resources of the calculation unit by utilizing a dedicated storage unit. In this case, the capacity allocated to the storage unit can be determined depending on the required throughput and disparity search range.

[0058] <Embodiment 2> FIG. 11 is a diagram showing the overall configuration of a distance measuring device 1100 according to a second embodiment of the present invention. The same components as those in the first embodiment are assigned the same reference numerals, and detailed description thereof will be omitted. The distance measuring device 1100 according to the second embodiment includes a first correlation calculation unit 1101 and a first parallax calculation unit 1102, and a second correlation calculation unit 1103 and a second parallax calculation unit 1104, instead of the correlation calculation unit 108 and the parallax calculation unit 109 of the distance measuring device 100 according to the first embodiment. The distance measuring device 1100 according to the second embodiment performs known hierarchical processing using the first correlation calculation unit 1101 and the first parallax calculation unit 1102, and the second correlation calculation unit 1103 and the second parallax calculation unit 1104. The first correlation calculation unit 1101 and the first parallax calculation unit 1102 are examples of external adjustment means that externally adjust addresses read from the memory. The first correlation calculation unit 1101 is an example of first correlation acquisition means. The first parallax calculation unit 1102 is an example of a first parallax amount calculation means. The second correlation calculation unit 1103 is an example of a second correlation acquisition means that acquires correlation information between the first image and the second image by narrowing the target range more than the first correlation acquisition means in accordance with the calculation result of the first parallax amount calculation means. The second parallax calculation unit 1104 is an example of a second parallax amount calculation means that calculates the amount of parallax between the image of the standard region and the image of the reference region based on the correlation information acquired by the second correlation acquisition means.

[0059] The first correlation calculation unit 1101 and the first parallax calculation unit 1102 perform template matching after reducing the image to simplify processing, for example, to calculate an approximate parallax. The second correlation calculation unit 1103 and the second parallax calculation unit 1104 receive the calculation result of the first parallax calculation unit 1102 and perform detailed template matching and parallax calculation only in the vicinity of the parallax position.

[0060] The template matching performed by the first correlation calculation unit 1101 may be a known method or the method described in embodiment 1. Embodiment 2 is an example in which the method according to the present invention is applied to the template matching performed by the second correlation calculation unit 1103. Details thereof will be described below.

[0061] The control flow instructed by the integrated control unit 111 according to the second embodiment, which illustrates write control to the line buffer for the second correlation calculation unit 1103, is the same as Fig. 9 shown in the first embodiment. The number of search ranges to be performed in the detailed template matching performed by the second correlation calculation unit 1103 is associated with the number of connected pixel data.

[0062] 12 is a diagram showing read control to a line buffer for the second correlation calculation unit 1103 in the control flow instructed by the integrated control unit 111 according to the second embodiment. In the processing in FIG. 12, the same processes as those in FIG. 10 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0063] In the process of FIG. 12, the parallax amount is received from the first parallax amount calculation unit 1102 and correlation calculation is performed only on a small area in the vicinity thereof, so unlike the process of FIG. 10, there is no loop regarding the search area 203.

[0064] In step S1200, the second correlation calculation unit 1103 receives from the first parallax calculation unit 1102 the amount of parallax calculated by the first parallax calculation unit 1102, that is, the first amount of parallax corresponding to the reference position 200a.

[0065] In step S1201, the second correlation calculation unit 1103 calculates a read address b for the right image. This calculation corrects for any deviation from the reference position 200a, as in the case of calculating the read address a, and then determines the address to be read according to the first parallax amount. The partial search area 700 read using the address determined here corresponds to the small area where template matching is performed in the subsequent stage of hierarchical processing. The processing in step S1107 in Figure 12 is an example of processing for calculating a correlation for an address adjusted by external adjustment means that externally adjusts the address read from the memory.

[0066] As described above, in this embodiment, the distance measuring device 1100, which calculates distance by performing template matching, performs hierarchical calculation of the amount of parallax. At this time, by assigning a capacity to the storage unit to correspond to the search range required by the second correlation calculation unit 1103, it is possible to achieve high throughput while reducing the register resources of the calculation unit.

[0067] (others) The present invention is not limited to the above examples, and various embodiments are possible without departing from the spirit of the present invention. For example, the imaging units 101a and 101b may share a portion of the imaging elements, or an imaging element having pixels for distance measurement may be used. In the above-described embodiments, SSD is used as a method for calculating correlation by template matching, but other correlation calculation methods may be used as long as they are processes that require reference to the periphery of the target position. Furthermore, other processes that similarly refer to the periphery of the target position, such as filter processing, may also be used, in addition to parallax calculation.

[0068] Note that a computer program that realizes part of the control in each embodiment and the functions of the above-described examples may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0069] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0070] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention.

[0071] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image acquisition means for acquiring a first image and a second image having parallax; a correlation acquisition means for acquiring a plurality of pieces of correlation information between an image of a reference area in the first image and images of a plurality of reference areas in the second image corresponding to the reference area; a parallax amount calculation means for calculating a parallax amount between the image of the standard area and the image of the reference area based on the plurality of pieces of correlation information; a storage means for storing pixel data constituting the first image and the second image in a memory for a predetermined period of time; and the correlation acquisition means sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means, concatenates the acquired pixel data with previous pixel data each time the pixel data is acquired, and stores the concatenated pixel data for each address in the memory; 1. An image processing device comprising: (Configuration 2) the correlation acquisition means calculates the correlation by template matching; 2. The image processing device according to claim 1, wherein: (Configuration 3) the correlation acquisition means reads out the connected pixel data contained in one address of the memory, and calculates one or more correlations using the connected pixel data. 3. The image processing device according to configuration 1 or 2, (Configuration 4) the correlation acquisition means has an adjustment means for adjusting an address read from the memory, and performs multiple reads to thereby calculate multiple correlations; 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) the correlation acquisition means is a first correlation acquisition means, the parallax amount calculation means is a first parallax amount calculation means, a second correlation acquisition means for acquiring correlation information between the first image and the second image by narrowing a target range compared to that of the first correlation acquisition means in accordance with a calculation result of the first parallax amount calculation means; a second parallax amount calculation means for calculating a parallax amount between the image of the standard area and the image of the reference area based on the correlation information acquired by the second correlation acquisition means; further comprising 5. The image processing device according to any one of configurations 1 to 4. (Method 1) an image acquisition step of acquiring a first image and a second image having parallax; a correlation acquisition step of acquiring a plurality of pieces of correlation information between an image of a reference area in the first image and images of a plurality of reference areas in the second image corresponding to the reference area; a parallax amount calculation step of calculating a parallax amount between the image of the standard area and the image of the reference area based on the plurality of pieces of correlation information; a storage step of storing pixel data constituting the first image and the second image in a memory for a predetermined period of time; and the correlation acquisition step sequentially acquires pixel data constituting the first image and the second image acquired in the image acquisition step, concatenates the acquired pixel data with previous pixel data each time the pixel data is acquired, and stores the concatenated pixel data for each address in the memory; 2. A method for controlling an image processing apparatus comprising: (Program 1) Computer, an image acquisition means for acquiring a first image and a second image having parallax; a correlation acquisition means for acquiring a plurality of pieces of correlation information between an image of a reference area in the first image and images of a plurality of reference areas corresponding to the reference area in the second image; a parallax amount calculation means for calculating a parallax amount between the image of the standard area and the image of the reference area based on the plurality of pieces of correlation information; and a storage means for storing pixel data constituting the first image and the second image in a memory for a predetermined period of time; It functions as the correlation acquisition means sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means, concatenates the acquired pixel data with previous pixel data each time the pixel data is acquired, and stores the concatenated pixel data for each address in the memory; A program characterized by: [Explanation of symbols]

[0072] 100 Rangefinder 101a, 101b imaging unit 102 Range image generation unit 103a, 103b lenses 104a, 104b imaging elements 105a, 105b Image transmission unit 106 Image receiving unit 107 Image correction unit 108 Correlation calculation unit 109 Parallax amount calculation unit 110 Distance calculation unit 111 General control unit 112 Storage section

Claims

1. an image acquisition means for acquiring a first image and a second image having parallax; a correlation acquisition means for acquiring a plurality of pieces of correlation information between an image of a reference area in the first image and images of a plurality of reference areas in the second image corresponding to the reference area; a parallax amount calculation means for calculating a parallax amount between the image of the standard area and the image of the reference area based on the plurality of pieces of correlation information; a storage means for storing pixel data constituting the first image and the second image in a memory for a predetermined period of time; and the correlation acquisition means sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means, concatenates the acquired pixel data with previous pixel data each time the pixel data is acquired, and stores the concatenated pixel data for each address in the memory; 1. An image processing device comprising:

2. the correlation acquisition means calculates the correlation by template matching; 2. The image processing device according to claim 1, wherein:

3. the correlation acquisition means reads out the connected pixel data contained in one address of the memory, and calculates one or more correlations using the connected pixel data.

2. The image processing device according to claim 1, wherein:

4. the correlation acquisition means has an adjustment means for adjusting an address read from the memory, and performs multiple reads to thereby calculate multiple correlations; 2. The image processing device according to claim 1, wherein:

5. the correlation acquisition means is a first correlation acquisition means, the parallax amount calculation means is a first parallax amount calculation means, a second correlation acquisition means for acquiring correlation information between the first image and the second image by narrowing a target range compared to that of the first correlation acquisition means in accordance with a calculation result of the first parallax amount calculation means; a second parallax amount calculation means for calculating a parallax amount between the image of the standard area and the image of the reference area based on the correlation information acquired by the second correlation acquisition means; further comprising 2. The image processing device according to claim 1, wherein:

6. an image acquisition step of acquiring a first image and a second image having parallax; a correlation acquisition step of acquiring a plurality of pieces of correlation information between an image of a reference area in the first image and images of a plurality of reference areas in the second image corresponding to the reference area; a parallax amount calculation step of calculating a parallax amount between the image of the standard area and the image of the reference area based on the plurality of pieces of correlation information; a storage step of storing pixel data constituting the first image and the second image in a memory for a predetermined period of time; and the correlation acquisition step sequentially acquires pixel data constituting the first image and the second image acquired in the image acquisition step, concatenates the acquired pixel data with previous pixel data each time the pixel data is acquired, and stores the concatenated pixel data for each address in the memory; 2. A method for controlling an image processing apparatus comprising:

7. Computer, an image acquisition means for acquiring a first image and a second image having parallax; a correlation acquisition means for acquiring a plurality of pieces of correlation information between an image of a reference area in the first image and images of a plurality of reference areas corresponding to the reference area in the second image; a parallax amount calculation means for calculating a parallax amount between the image of the standard area and the image of the reference area based on the plurality of pieces of correlation information; and a storage means for storing pixel data constituting the first image and the second image in a memory for a predetermined period of time; It functions as the correlation acquisition means sequentially acquires pixel data constituting the first image and the second image acquired by the image acquisition means, concatenates the acquired pixel data with previous pixel data each time the pixel data is acquired, and stores the concatenated pixel data for each address in the memory; A program characterized by:

Images