Image processing device and image processing method

Abstract

To increase a possibility of parallax calculation even when invalidity of a higher-order parallax is determined in hierarchical search.SOLUTION: An image processing device for determining parallax by comparing images captured by two cameras includes: first hierarchical matching processing for obtaining first parallax in image region units of a first size by identifying an image block which has a high degree of similarity to a first image block contained in one of the two images from a first search range determined on the basis of a predetermined first search width in the other image; and second hierarchical matching processing for obtaining second parallax in image region units of a second size by identifying the same from a second search range determined on the basis of the first parallax and a predetermined second search width in the other image. The image processing device performs a search by broadening the first search width or the second search range when an image block which has a high degree of similarity to the first image block exists, at an end within the first search range during the first hierarchical matching processing.SELECTED DRAWING: Figure 3

Description

Technical Field

[0006] , , , , ,

[0001] The present invention relates to an image processing apparatus and an image processing method for monitoring an object existing around a vehicle by using an image captured by an imaging device.

Background Art

[0002] There is known a position measurement device that measures the distance to an object existing within the camera's field of view based on a stereo image captured by a stereo camera having a pair of cameras, using the principle of triangulation. That is, the principle of triangulation calculates the distance from the camera to the object by using the displacement (parallax) of the positions of the images of the same object captured by the left and right cameras. The derivation of parallax is realized by specifying where the image of an object on one image exists on the other image.

[0003] <​​​​​​​​​​​​​​​​​ [Patent Document 1] Japanese Patent Publication No. 2001-319229 [Overview of the project] [Problems that the invention aims to solve]

[0007] Patent Document 1 attempts to reduce computation time and complexity by searching only around pre-calculated low-resolution (high-resolution) parallax in order to calculate higher-resolution (lower-resolution) parallax. On the other hand, when searching for parallax, the reliability of the calculated parallax may be considered low and the parallax may be invalidated, for example, if the dissimilarity of the image regions of the left and right images is minimized at the edge of the parallax search range, and it is expected that the true parallax exists outside the search range. In hierarchical search, if a lower-resolution parallax is considered invalid, the corresponding high-resolution parallax cannot be searched and all become invalid. Therefore, there is a problem that a large number of invalid parallaxes may be generated in the final high-resolution parallax image.

[0008] Given the above situation, there was a need for a method that, when a higher-level disparity is determined to be invalid in hierarchical search, avoids invalidating all corresponding lower-level disparities, thereby increasing the likelihood of disparity calculation. [Means for solving the problem]

[0009] To solve the above problems, one aspect of the present invention is an image processing apparatus that includes a disparity calculation unit that compares two images captured by two cameras to determine the disparity. The disparity calculation unit performs a first-level stereo matching process to determine the first disparity in units of an image region of first size by identifying the image block with the highest similarity to the first image block, which is an image region of first size contained in one of the two images, from a first search range determined based on a predetermined first search width in the other image of the two images, and a second-level stereo matching process to determine the second disparity in units of an image region of second size by identifying the image block with the highest similarity to the second image block, which is an image region of second size smaller than the first size contained in one of the two images, from a second search range determined based on the first disparity and a predetermined second search width in the other image of the two images. In the first-level stereo matching process, if there is an image block with the highest similarity to the first image block at the edge of the first search range, the disparity calculation unit expands the first search range or the second search range and performs the search. Here, the search range is the information of the start and end points of the search, which determines the search width. [Effects of the Invention]

[0010] According to at least one aspect of the present invention, when a higher-level disparity is determined to be invalid in a hierarchical search, it is possible to avoid invalidating all corresponding lower-level disparities and to increase the likelihood of disparity calculation. Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the hardware configuration of the control system inside the vehicle. [Figure 2] This figure shows a comparison of the surrounding area of ​​the vehicle, a coarse-range image, and a detailed-range image. [Figure 3] This is a block diagram showing an example of the functional configuration of a computing device according to the first embodiment of the present invention. [Figure 4] This figure shows the concept of disparity determination by the disparity search unit included in the disparity calculation unit in the first embodiment of the present invention. [Figure 5] This figure shows an example of determining local research and an example of the direction of extending the search range during research according to the first embodiment of the present invention. [Figure 6] This flowchart shows an example of the procedure for local search processing at the Nth hierarchical level according to the first embodiment of the present invention. [Figure 7] This is a block diagram showing an example of the functional configuration of the disparity calculation unit in a computing device according to a second embodiment of the present invention. [Figure 8] This flowchart shows an example of the procedure for local search processing at the Nth hierarchical level according to a second embodiment of the present invention. [Figure 9] This figure shows examples of the magnitude of dissimilarity within the local search range in the second embodiment of the present invention. [Figure 10] This flowchart shows an example of the procedure for local search processing at the Nth hierarchical level according to a third embodiment of the present invention. [Modes for carrying out the invention]

[0012] Hereinafter, examples of embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, identical components or components having substantially the same function are denoted by the same reference numeral, and redundant descriptions are omitted. Furthermore, if there are multiple components having the same or similar function, different subscripts may be used with the same reference numeral in the description. In addition, if it is not necessary to distinguish between these multiple components, the subscript may be omitted in the description.

[0013] <First Embodiment> First, an arithmetic device (an example of an image processing device) according to the first embodiment of the present invention will be described with reference to Figures 1 to 6.

[0014] FIG. 1 is a diagram showing an example of the hardware configuration of a control system in a vehicle. The vehicle 10 shown in FIG. 1 includes an arithmetic unit 1, a left camera 2, a right camera 3, and a vehicle control device 4. The arithmetic unit 1 and the vehicle control device 4 are electronic control units (ECUs). The left camera 2 and the right camera 3 output captured images obtained by capturing the surroundings of the vehicle 10 to the arithmetic unit 1. The left camera 2 and the right camera 3 are arranged side by side in the horizontal direction. Hereinafter, the captured image of the left camera 2 will be referred to as the first camera image 21, and the captured image of the right camera 3 will be referred to as the second camera image 31. Also, the left camera 2 may be referred to as "the first camera", and the right camera 3 may be referred to as "the second camera".

[0015] Since the left camera 2 and the right camera 3 are arranged so as to face the horizontal direction, both the first camera image 21 (left image) and the second camera image 31 (right image) are images obtained by capturing a three-dimensional space that spreads in the vertical and horizontal directions. However, the horizontal and vertical here do not have a strict meaning, but rather mean that the left-right direction of the first camera image 21 and the second camera image 31 corresponds to the horizontal direction rather than the top-bottom direction. For example, the left camera 2 and the right camera 3 may be tilted by up to about 30 degrees around the optical axis as the rotation center.

[0016] The arithmetic unit 1 includes a CPU 11 which is a central processing unit, a ROM 12 which is a read-only storage device, a RAM 13 which is a readable and writable storage device, and an interface 14 which receives the first camera image 21 and the second camera image 31. By the CPU 11 expanding and executing the computer program stored in the ROM 12 in the RAM 13, a plurality of functions of the arithmetic unit 1 described later are realized. Instead of the ROM 12, a computer program may be stored in a non-volatile storage (not shown). The ROM 12 and the non-volatile storage are used as an example of a computer-readable non-transitory recording medium storing the program.

[0017] However, instead of the combination of the CPU 11, ROM 12, and RAM 13, the arithmetic unit 1 may implement some or all of their functions using an FPGA (Field Programmable Gate Array), which is a rewritable logic circuit, or an ASIC (Application Specific Integrated Circuit), which is an integrated circuit for specific applications. Also, instead of the combination of the CPU 11, ROM 12, and RAM 13, the arithmetic unit 1 may be implemented by a combination of different configurations, for example, a combination of the CPU 11, ROM 12, RAM 13, and FPGA.

[0018] The interface 14 receives the first camera image 21 (left image) and the second camera image 31 (right image) and provides them to the CPU 11. However, since only a small amount of information can be stored in the registers provided in the CPU 11, the interface 14 stores the received first camera image 21 and second camera image 31 in the RAM 13 so that the CPU 11 can freely refer to that information.

[0019] The arithmetic unit 1 creates a distance image 41 (see FIG. 2) including distance information to an object around the vehicle 10 based on parallax using the input first camera image 21 (left image) and second camera image 31 (right image), and outputs it to the vehicle control device 4. The process by which the arithmetic unit 1 creates the distance image 41 will be described in detail later by referring to FIG. 2 and subsequent figures. The distance image 41 represents the distance perspective in a two-dimensional plane using color shading or color saturation. However, in the present embodiment, integer values from "0" to "255" representing shading are stored in each pixel of the distance image 41. Among these, "0" is a special numerical value indicating that distance calculation was impossible.

[0020] The range shown in the distance image 41 is approximately the same as that of the first camera image 21 (left image) and the second camera image 31 (right image). However, the range shown in the distance image 41 may be narrower than that shown in the first camera image 21 and the second camera image 31. The number of pixels in the distance image 41 may be approximately the same as that of the first camera image 21 and the second camera image 31, or it may be less than that of the first camera image 21 and the second camera image 31.

[0021] In other words, the three-dimensional spaces recorded in the first camera image 21 (left image) and the second camera image 31 (right image) only need to overlap in some part. The mounting positions and orientations of the left camera 2 and the right camera 3 on the vehicle 10 are known, and the relative positions of the first camera image 21 and the second camera image 31 are also known. Therefore, the position in the three-dimensional space of a given pixel in the first camera image 21 corresponds to a position in the second camera image 31, which is also known. The relative positions of the first camera image 21 and the second camera image 31 may be determined by calculation, or they may be determined by referring to a lookup table or the like prepared in advance.

[0022] In other words, the relative positions of the left camera 2 and the right camera 3 are known, and the intrinsic parameters of the two cameras are also known, so the relationship between the distance of the subject to the camera and the parallax is also known. In this embodiment, the relationship between distance and parallax is stored in the ROM 12 of the arithmetic unit 1 as a parallax table (not shown). In this embodiment, since the relationship between distance and parallax is known, parallax can also be called "information corresponding to distance". Furthermore, in this embodiment, distance and parallax can be easily converted to each other, so they can be considered to be substantially the same information.

[0023] The vehicle control device 4 controls the vehicle 10 using a distance image 41 input from the arithmetic unit 1. For example, the vehicle control device 4 detects obstacles present around the vehicle 10 from the distance image 41 and controls the vehicle 10 to avoid collisions with the detected obstacles. The control of the vehicle 10 by the vehicle control device 4 is achieved, for example, by the vehicle control device 4 outputting operation commands to a steering device (not shown) and a braking device (not shown).

[0024] [Images showing the area around the vehicle, including rough distance images and detailed distance images] Figure 2 shows a comparison of the surrounding area of ​​the subject, a coarse-range image, and a detailed-range image. Figure 2 illustrates a comparison of the resolution of the information stored in the first camera image 21, the second camera image 31, the coarse depth image 45, and the detailed depth image 46. However, the resolution referred to here is not the depth, but rather the resolution in the vertical and horizontal directions of the image. The coarse depth image 45 and the detailed depth image 46 are examples of the depth image 41 shown in Figure 1.

[0025] The upper part of Figure 2 shows the surroundings of vehicle 10, with other vehicles 20 present. The middle part of Figure 2 is a conceptual diagram showing the resolution of the coarse-length image 45 taken of vehicle 20, and the lower part of Figure 2 is a conceptual diagram showing the resolution of the detailed-length image 46, which has a higher resolution than the coarse-length image 45.

[0026] The coarse distance image 45 shown in the middle of Figure 2 stores distance information for a total of 25 image regions, obtained by dividing the region shown in the upper part of Figure 2 into 5 sections vertically and horizontally.

[0027] The detailed distance image 46 shown in the lower part of Figure 2 stores distance information for a total of 100 image regions, obtained by dividing the region shown in the upper part of Figure 2 into 10 vertical and 10 horizontal sections. In the coarse distance image 45 and the detailed distance image 46, the top and bottom of the illustrated image correspond to the vertical direction, and the left and right of the illustrated image correspond to the horizontal direction.

[0028] In this embodiment, as shown in the middle of Figure 2, the image region indicated by one distance information contained in the coarse distance image 45 is called the "first image block," and as shown in the lower part of Figure 2, the image region indicated by one distance information contained in the detailed distance image 46 is called the "second image block." The size of the first image block (first size) is larger than the size of the second image block (second size). In the example shown in Figure 2 and in this embodiment, the first image block is twice as wide and as high as the second image block, and its area is four times larger. However, it is sufficient that the first image block is larger than the second image block; the multiple is not particularly limited.

[0029] Note that the distance information for a single point is not necessarily obtained by measuring the entire first image block, but in this embodiment, that distance information is conveniently considered to represent the entire first image block. Furthermore, it is known which region of the first image block in the coarse distance image 45 each second image block in the detailed distance image 46 is contained within.

[0030] Each of the first camera image 21 and the second camera image 31 has brightness and saturation information with a resolution equal to or greater than that of the second image block. Referring to the example in Figure 2, each of the first camera image 21 and the second camera image 31 has at least 10 pixels vertically and horizontally, i.e., 100 pixels or more in total, within the range shown in the upper part of Figure 2. The first camera image 21 and the second camera image 31 may have approximately the same number of pixels as the detailed distance image 46, or they may have twice or ten times the number of pixels in the vertical or horizontal direction. In this embodiment, as an example, the first camera image 21 and the second camera image 31 have approximately the same number of pixels as the detailed distance image 46.

[0031] [Functional Configuration of the Parallax Calculation Unit] Next, an example of the functional configuration of the computing device 1 according to the first embodiment of the present invention will be described with reference to Figure 3. Figure 3 is a block diagram showing an example of the functional configuration of the arithmetic unit 1. As shown in Figure 3, the arithmetic unit 1 includes a disparity calculation unit 100, which consists of a disparity search unit 101, a higher-level reference disparity determination unit 102, and a disparity invalidity determination unit 103. The functions of the disparity search unit 101, the higher-level reference disparity determination unit 102, and the disparity invalidity determination unit 103 of the disparity calculation unit 100 are realized by software processing by the CPU 11 described above. The information generated by the processing described below, such as search range information, disparity information, dissimilarity information, higher-level disparity information, disparity invalidity information, and search range change instructions, are temporarily stored in RAM 13.

[0032] Furthermore, as previously described, the distance from the left and right cameras to the subject can be determined based on the parallax of two camera images containing the same subject, using the principle of triangulation. Therefore, the parallax calculation unit 100 may generate distance images (for example, a coarse distance image 45, a detailed distance image 46, etc.) based on the parallax information of the two images and output them to the vehicle control device 4. However, in this embodiment, an example in which the parallax calculation unit 100 calculates the parallax information and outputs it to the vehicle control device 4 will be described.

[0033] The parallax calculation unit 100 receives the first camera image 21 (left image) and the second camera image 31 (right image), which are captured from different viewpoints by the left camera 21 and the right camera 31.

[0034] The parallax search unit 101 performs a parallax search (parallax determination) by comparing two images, the first camera image 21 and the second camera image 31, and outputs "parallax information" and "dissimilarity information" used for parallax determination as search results. Stereo matching processing can be used as an example for this search process. In stereo matching processing, the comparison target image blocks of the two images are shifted by one pixel each and compared. The stereo matching processing described here is an example of first-level stereo matching processing. If the parallax generated at this time is not the highest-level parallax in hierarchical search (see, for example, Patent Document 1), the parallax search unit 101 obtains "search range information" from the higher-level reference parallax determination unit 102 and performs a local search that searches only within that range. This local search processing is an example of second-level stereo matching processing. In the following explanation, parallax search will also be simply referred to as "search".

[0035] The above search range information is the information of the starting and ending pixels in another camera image (e.g., second camera image 31) relative to a reference camera image (e.g., first camera image 21) in disparity search using stereo matching processing, which determines the search range (pixel shift amount). The search width can be calculated from the starting and ending points. Generally, the higher the resolution of the image, the wider the search width.

[0036] Furthermore, the highest-level parallax is the parallax searched from the lowest-resolution image when multiple hierarchical resolution images are generated from the camera image (original image). As a result of the search, the parallax search unit 101 determines the parallax so that the dissimilarity is minimized. The determined parallax information (information representing the position of the parallax on the image) is sent to the parallax invalidity determination unit 103 along with the dissimilarity information.

[0037] The parallax invalidity determination unit 103 refers to the "dissimilarity information" output from the parallax search unit 101 and determines whether the parallax determined by the parallax search unit 101 is located at the edge of the local search range. If the determined parallax is at the edge, the parallax invalidity determination unit 103 considers that there is a high probability that a point with even lower dissimilarity, i.e., a true parallax, exists outside the local search range, and instructs the parallax search unit 101 to change (for example, extend) the local search range to outside the edge of the local search range where the determined parallax is located and to re-search (indicated as "search range change instruction" in the figure).

[0038] Regarding the extent of extension of the local search range, the extended search range and extension width are predetermined for each condition such as the image layer (resolution), and registered in the disparity calculation unit 100 (for example, ROM 12). The extent of this extension of the local search range differs for each image layer, for example, depending on the allowable search time and cost. For example, the search time differs depending on the processing power of the arithmetic unit 1 and the data size of the camera image. The maximum extension width of the local search range is determined by the circuit size (processing power) of the arithmetic unit 1.

[0039] The parallax search unit 101, upon receiving a search range change instruction from the parallax invalidity determination unit 103, performs a re-search and outputs the point with the minimum dissimilarity, including the extended search range, as the parallax.

[0040] In this case, an upper limit may be set on the extended search range, taking into consideration the computational load and computation time of the arithmetic unit 1. That is, when the arithmetic unit 1 expands the first search width (search width of the upper hierarchy) or the second search range (search range of the lower hierarchy) to perform a search, an upper limit is set in advance on the first search width or the second search range. With this configuration, by setting an upper limit on the first search width or the second search range, it becomes possible to reduce the computational load and computation time of the arithmetic unit 1.

[0041] Furthermore, the parallax search unit 101 may terminate the search once it finds the first local minimum of dissimilarity during the re-search process, and output the point of the first local minimum as the parallax. In other words, when the parallax search unit 101 expands the first search width (search width of the upper level) or the second search range (search range of the lower level) to perform a search, it can be configured to terminate the search based on a point where the similarity decreases along the search direction. With this configuration, the parallax can be determined based on a point where the similarity decreases along the search direction, thereby reducing the computation amount of the arithmetic unit 1 and shortening the computation time.

[0042] In this specification, the terms "minimum" and "local minimum" are used, but this specification includes not only minimums and local minimums in the strictly mathematical sense, but also practical minimums and local minimums. For example, a local minimum is a point where the derivative is 0 or less than or equal to a predetermined value (for example, a small value close to 0).

[0043] Furthermore, the presence or absence of extended search and the extended range may be changed depending on whether the extension direction of the local search range, as explained later using Figure 5, is the same as or opposite to the direction of the search. That is, the disparity search unit 101 can be configured to determine the first search width or the second search range based on which end of the first search range the image block with the highest similarity was found when, in the first-level stereo matching process (a higher-level local search), an image block with the highest similarity to the first image block exists at the end of the first search range. With this configuration, by determining the first search width or the second search range based on which end of the first search range the image block with the highest similarity (minimum dissimilarity) was found, it is possible to search for disparities with higher similarity.

[0044] The "parallax information" and "dissimilarity information" from the re-search results are again sent from the parallax search unit 101 to the parallax invalidity determination unit 103. Here, the parallax invalidity determination unit 103 may send the results of the re-search performed once by the parallax search unit 101 as "parallax information" and "parallax invalidity (valid / invalid) information" to the vehicle control device 4, or it may perform the determination for re-search again in the parallax invalidity determination unit 103. As a basic concept of valid / invalid, if the position of the determined parallax is not at the edge of the local search range, the parallax is determined to be valid, and if the position of the determined parallax is at the edge of the local search range, the parallax is determined to be invalid.

[0045] [Disparity determination by the disparity search unit] Next, the concept of disparity determination by the disparity search unit 101 (Figure 3) included in the first embodiment of the present invention will be explained with reference to Figure 4. Figure 4 is a diagram illustrating the concept of disparity determination by the disparity search unit 101 of the disparity calculation unit 100. The upper part of Figure 4 shows the first camera image 21 and the second camera image 31. In this example, the system searches for a reference image, i.e., the image region of the second camera image 31, which corresponds to the image region 201 of the first camera image 21, which is the reference image. Although the first camera image 21 is used as the reference image here, the second camera image 31 may also be used as the reference image.

[0046] In this embodiment, since the left camera 2 and the right camera 3 are installed side by side horizontally, the epipolar line 202 due to epipolar geometry is shown as a dashed line extending horizontally. When the horizontal width of the first camera image 21 and the second camera image 31 is 1000 pixels, and the center of the image region 201 is the 400th pixel from the left edge of the first camera image 21, the horizontal values ​​of the reference image, the second camera image 31, are assigned values ​​from -400 to +600 as shown. For example, if the disparity search unit 101 determines that the image region 203 centered at the 500th pixel from the left edge of the reference image corresponds to the image region 201 of the reference image, the disparity is calculated to be "100".

[0047] The middle section of Figure 4 shows an example of the dissimilarity between the reference image (second camera image 31) and the image region 201 at each position on the epipolar line 202. In the example shown in the middle section of Figure 4, there is a peak (local minimum) where the dissimilarity is smallest, and there are no other peaks with similar values, so the disparity search unit 101 calculates the disparity of the image region 201 as "100".

[0048] The lower part of Figure 4 shows another example of the dissimilarity between the reference image (second camera image 31) and the image region 201 at each position on the epipolar line 202. In the example shown in the lower part of Figure 4, there are two peaks with the minimum dissimilarity, and their dissimilarity values ​​are approximately the same, so it is not possible to determine whether the parallax of the image region 201 is "-180" or "+115". In such cases, the parallax search unit 101 sets the image region 201 as an invalid region and sets the parallax to "0" at the position in the distance image 41 corresponding to the image region 201. A related invention for the case where there are multiple peaks with the minimum dissimilarity is shown in the third embodiment described later.

[0049] [Examples of local research determination and direction of search range extension during research] Figure 5 shows an example of local research determination and an example of the search range extension direction during research according to the first embodiment of the present invention. In the graph in the figure, the horizontal axis represents the position (corresponding to the amount of pixel shift) when comparing both the first camera image 21 and the second camera image 31, and the vertical axis represents the dissimilarity of the comparison result. The region enclosed by the two dashed lines indicates the local search range.

[0050] In the upper graph of Figure 5, the dissimilarity of the parallax within the local search range decreases monotonically, with the dissimilarity being minimized at the right edge of the local search range. Therefore, the parallax search unit 101 outputs the parallax 501 located at the right edge of the local search range, but at the same time outputs the five dissimilarity values ​​calculated here (including the dissimilarity value corresponding to parallax 501). The parallax invalidity determination unit 103 obtains the parallax 501 along with the five dissimilarity values ​​from the parallax search unit 101 and can determine that the parallax 501 is located at the right edge of the local search range, i.e., that the parallax 501 is invalid. Therefore, the parallax 501 becomes subject to a search range change instruction by the parallax invalidity determination unit 103.

[0051] Alternatively, the parallax search unit 101 may output, instead of dissimilarity, that the output parallax is at the edge or right edge of the local search range. The parallax invalidity determination unit 103 can determine from this information that the parallax output by the parallax search unit 101 is at the edge (in this case, the right edge) of the local search range. In the example shown in the upper part of Figure 5, the direction of extension of the search range during re-search is to the right (the direction in which the amount of pixel shift increases).

[0052] In the graph in the middle of Figure 5, the dissimilarity of the parallax within the local search range does not decrease monotonically but has a maximum value. However, the parallax search unit 101 still outputs the parallax 502 located at the right end of the local search range as the point of minimum dissimilarity. Therefore, the parallax 502 becomes the target of the search range change instruction by the parallax invalidity determination unit 103. The direction of extension of the search range during re-search is to the right (the direction in which the amount of pixel shift increases).

[0053] The graph in the lower part of Figure 5 shows that the dissimilarity of parallax within the local search range does not increase monotonically but has local maximums and minimums, with the dissimilarity of parallax 503, located at the leftmost edge of the local search range, being the minimum. Therefore, parallax 503 becomes the target of the search range change instruction by the parallax invalidity determination unit 103. The direction of extension of the search range during re-search is to the left (the direction in which the amount of pixel shift decreases).

[0054] [Example of local search procedure] Next, an example of the procedure for local search processing of the Nth hierarchical search in the first embodiment of the present invention will be described with reference to Figure 6. Figure 6 is a flowchart showing an example of the procedure for local search processing of the Nth hierarchical search by the disparity calculation unit 100 of the first embodiment.

[0055] First, in the parallax calculation unit 100, the higher-level reference parallax determination unit 102 refers to the parallax of the (N-1)th layer (lower resolution) which is higher than the Nth layer obtained from the parallax invalidity determination unit 103 (step S1) and determines the initial search range in the Nth layer (step S2). If the Nth layer is the highest layer (lowest resolution), the search range is set in advance and stored in the ROM 12.

[0056] Next, the disparity search unit 101 sets the initial search range determined by the higher-level reference disparity determination unit 102 as the search range for local search, and performs a local search within that search range (step S3).

[0057] Next, the parallax invalidity determination unit 103 determines, based on the search results from the parallax search unit 101, whether the point with the minimum dissimilarity is located at the edge of the search range (step S4).

[0058] If the point with the minimum dissimilarity in step S4 is not at the edge of the search range (NO in step S4), the parallax invalidity determination unit 103 determines and outputs the parallax and whether the parallax is enabled or disabled (in this case, enabled) (step S7). The parallax and parallax enabled / disabled information is output to the vehicle control device 4 and the higher-level reference parallax determination unit 102.

[0059] On the other hand, if the point with the minimum dissimilarity is at the edge of the search range (YES in step S4), the parallax invalidity determination unit 103 determines whether the number of re-searches is within the preset upper limit of the number of re-searches (step S5). If the number of re-searches is within the preset upper limit of the number of re-searches (YES in step S5), the parallax invalidity determination unit 103 issues a search range change instruction to the parallax search unit 101. Note that the determination process in step S5 is provided considering the allowable processing time, but it may be deleted.

[0060] Next, when the disparity search unit 101 receives a search range change instruction, it corrects the search range during the local search in step S3 (step S6) and re-searches for disparity (step S3).

[0061] Next, after determining YES in step S4, if the parallax invalidity determination unit 103 determines that the number of re-searches has reached the preset upper limit of the number of re-searches (NO in step S5), it confirms and outputs the parallax and whether the parallax is enabled or disabled (step S7). After processing in step S7, the local search process of the Nth hierarchical search is terminated.

[0062] As described above, the image processing apparatus (arithmetic unit 1) according to the first embodiment is an image processing apparatus equipped with a disparity calculation unit (disparity calculation unit 100) that compares two images (first camera image 21, second camera image 31) captured by two cameras (left camera 2, right camera 3) to determine the disparity. The disparity calculation unit performs the following: first-level stereo matching processing (upper-level local search) to find the first disparity in units of first-level image regions by identifying the image block (e.g., image region 203) that has the highest similarity to the first image block (e.g., image region 201) which is an image region of first size contained in one of the two images, from a first search range (local search range) determined based on a predetermined first search width in the other image of the two images; and second-level stereo matching processing (lower-level local search (research)) to find the second disparity in units of second-level image regions by identifying the image block that has the highest similarity to the second image block (e.g., image region 203) which is an image region of first size contained in one of the two images, from a second search range determined based on the first disparity and a predetermined second search width in the other image of the two images. Then, in the first-level stereo matching process, if there is an image block with the highest similarity to the first image block at the edge of the first search range, the disparity calculation unit expands the first search width or the second search range and performs a search (for example, based on the modified (extended) local search range).

[0063] According to this embodiment with the above configuration, when a higher-level parallax is determined to be invalid in a hierarchical search, it is possible to avoid invalidating all corresponding lower-level parallaxes, thereby increasing the probability of parallax calculation. Furthermore, since high-resolution distance information can be calculated without increasing the amount of computation or computation time required for parallax derivation, even distant objects can be accurately identified. Therefore, according to this embodiment, it is possible to detect objects that have fallen onto the road at a greater distance, and to understand road surface conditions such as holes and bumps, making this embodiment suitable for application to autonomous driving (AD) and advanced driver assistance systems (ADAS) of vehicles.

[0064] Furthermore, this embodiment is configured to re-search by changing the local search range (or search width) even if disparity is deemed invalid. Therefore, this embodiment not only finds the true minimum value (or local minimum value) of dissimilarity (or similarity) from two images, but also feeds back to the same or lower levels in the hierarchical search, enabling efficient searching for effective disparity without increasing the computational load even if an invalidity determination is made.

[0065] <Second Embodiment> The second embodiment is an example in which an invalidation judgment is output for a given level when the dissimilarity of the parallax reaches its minimum value at the edge of the local search range. The second embodiment will be described below, focusing mainly on the differences from the first embodiment.

[0066] [Functional Configuration of the Parallax Calculation Unit] Figure 7 is a block diagram showing an example of the functional configuration of the disparity calculation unit in a computing device (an example of an image processing device) according to a second embodiment of the present invention. As shown in Figure 7, the computing device 1 includes a disparity calculation unit 100A composed of a disparity search unit 101, a higher-level reference disparity determination unit 102, and a disparity invalidity determination unit 103. However, the disparity calculation unit 100A of the computing device 1 according to this embodiment differs from the disparity calculation unit 100 according to the first embodiment (Figure 3) in that it does not have a signal line that sends a search range change instruction from the higher-level reference disparity determination unit 102 to the disparity search unit 101.

[0067] Similar to the first embodiment, the parallax search unit 101 receives the first camera image 21 (left image) and the second camera image 31 (right image), which are captured from different viewpoints by the left camera 21 and the right camera 31.

[0068] The parallax search unit 101, similar to the first embodiment, performs parallax search (parallax determination) by comparing two images, the first camera image 21 and the second camera image 31, and outputs "parallax information" and "dissimilarity information" used for parallax determination as search results. This search process is an example of first-level stereo matching processing. If the parallax generated at this time is not the highest-level parallax in hierarchical search (see, for example, Patent Document 1), the parallax search unit 101 obtains "search range information" from the higher-level reference parallax determination unit 102 and performs a local search that searches only within that range. This local search process is an example of second-level stereo matching processing.

[0069] As a result of the search, the parallax search unit 101 determines the parallax such that the dissimilarity is minimized. The determined parallax information (information representing the position of the parallax on the image) is sent to the parallax invalidity determination unit 103 along with the dissimilarity information.

[0070] The parallax invalidity determination unit 103 refers to the "dissimilarity information" output from the parallax search unit 101 and determines whether the parallax determined by the parallax search unit 101 is located at the edge of the local search range. If the determined parallax is at the edge, the parallax invalidity determination unit 103 considers that there is a high probability that a point with even lower dissimilarity, i.e., a true parallax, exists outside the local search range, and determines that the parallax is invalid. The parallax invalidity determination unit 103 outputs the "parallax information" output from the parallax search unit 101 and the "parallax invalidity (valid / invalid) information" to the vehicle control device 4 and the higher-level reference parallax determination unit 102. Parallax determined to be invalid here is referenced as a higher-level parallax during local searches at lower levels, but if it remains an invalid parallax, all lower-level parallaxes will be determined to be invalid. In such cases, for example, by using the method described in Japanese Patent Publication No. 2020-140344, it may be possible to output the lower parallax while avoiding invalidation.

[0071] [Example of local search procedure] Next, an example of the procedure for local search processing of the Nth hierarchical search in the second embodiment of the present invention will be described with reference to Figure 8. Figure 8 is a flowchart showing an example of the procedure for local search processing of the Nth hierarchical search by the disparity calculation unit 100A in the second embodiment.

[0072] First, in the disparity calculation unit 100, the higher-level reference disparity determination unit 102 refers to the disparity of the N-1th layer (lower resolution), which is higher than the Nth layer obtained from the disparity invalidity determination unit 103 (step S11). Then, the higher-level reference disparity determination unit 102 determines the initial search range in the Nth layer.

[0073] Next, the higher-level reference parallax determination unit 102 determines whether the higher-level parallax is invalid or not (step S12). If the higher-level parallax is not invalid (NO in step S12), the unit proceeds to step S14.

[0074] If it is determined in step S12 that the higher-level disparity is invalid (YES in step S12), the higher-level reference disparity determination unit 102 switches the reference to a higher-level disparity adjacent to the higher-level disparity (step S13).

[0075] If the determination is NO in step S12, or after processing in step S13, the disparity search unit 101 sets the search range for local search. If the determination is NO in step S12, the disparity search unit 101 sets the initial search range determined by the higher-level reference disparity determination unit 102 as the search range for local search (step S14). Also, after processing in step S13, the disparity search unit 101 sets the search range for local search to the vicinity of the adjacent disparity of the higher level referenced by the higher-level reference disparity determination unit 102 (step S14).

[0076] Next, the parallax search unit 101 performs a local search within the set search range for the local search (step S15).

[0077] Next, the parallax invalidity determination unit 103 determines, based on the search results from the parallax search unit 101, whether the point with the minimum dissimilarity is located at the edge of the search range (step S16).

[0078] If the point with the minimum dissimilarity in step S16 is not at the edge of the search range (NO in step S16), the parallax invalidity determination unit 103 determines and outputs parallax and parallax validity (step S18).

[0079] On the other hand, if the point with the minimum dissimilarity is at the edge of the search range (YES in step S16), the parallax invalidity determination unit 103 determines that the parallax is invalid and outputs the determination result (step S17). Parallax and parallax valid / invalid information are output to the vehicle control device 4 and the higher-level reference parallax determination unit 102. After processing in step S17 or S18, the local search processing of the Nth hierarchical search is terminated.

[0080] In this embodiment, the process of determining whether the number of re-searches in step S5 of Figure 3, as described in the first embodiment, is within the upper limit may be provided after the processing in step S16.

[0081] Furthermore, the invalidation determination described in this embodiment does not need to be uniform for all local searches. For example, in this embodiment, in relatively high-level layers where a wide area beyond an arbitrary threshold is searched in a single disparity search due to low resolution (for example, the highest-level low-resolution layer among the three layers), the function to change the local search range when determining disparity invalidation in this invention (hereinafter referred to as "this function") can be turned off. Conversely, in relatively low-level layers where a narrow area is searched in a single search due to high resolution, this function can be turned on.

[0082] In this way, the arithmetic unit 1 (image processing device) can be configured to determine whether or not to expand the first search width (higher-level search width) or the second search range (lower-level search range) to perform a search based on the size of the first image block (image block of the higher-level camera image). According to this embodiment with such a configuration, disparity search can be performed efficiently according to the size of the first image block contained in one of the two images.

[0083] Furthermore, it is possible to operate the system in such a way that when the difference in dissimilarity within the local search is greater than an arbitrary threshold, it is assumed that there is a high probability that the minimum value exists outside the local search range, and this function is turned on. Conversely, when the difference in dissimilarity is small, it is assumed that there is a low probability that the minimum value exists outside the local search range, and this function is turned off.

[0084] In this configuration, the disparity search unit 101 in the arithmetic unit 1 (image processing device) can be configured to determine whether or not to expand the first search width or the second search range in the search in the first-level stereo matching process (local search of the higher level) based on whether or not the difference in similarity within the first search range is greater than a predetermined threshold. According to this embodiment with such a configuration, disparity search can be performed efficiently according to the difference in similarity within the first search range.

[0085] Furthermore, it is possible to operate this function in a way that enables it only when the dissimilarity of a parallax near the edge where the minimum value was obtained within the local search range differs from the dissimilarity of other parallaxes by a predetermined amount or more.

[0086] In this way, the arithmetic unit 1 (image processing device) can be configured to determine whether or not to expand the first search width or the second search range in the search in the first-level stereo matching process (local search of the higher level) based on whether or not the similarity of the image block with the highest similarity is greater than a predetermined threshold compared to the similarity of other image blocks within the search range. According to this embodiment with such a configuration, disparity search can be performed efficiently according to the difference in similarity within the first search range.

[0087] [Example of local search procedure] The following explanation of how to switch this function on and off, as described above, will be provided with reference to Figure 9. Figure 9 shows examples of the magnitude of dissimilarity within the local search range in the second embodiment of the present invention. For example, as shown in Figure 9, the smaller of the dissimilarity values ​​at the left and right ends of the local search range is defined as "LRmin," the maximum dissimilarity value excluding the left and right ends is defined as "SADmax," and the minimum is defined as "SADmin." Here, a threshold Th0 may be defined, and the disparity invalidity determination unit 103 may determine whether the following equation (1) is true and turn on this function when equation (1) is true.

[0088] [Mathematics 1] SADmin-LRmin ≧ Th0 (1)

[0089] This approach is adopted because the dissimilarity drops significantly at the left and right edges of the local search range, suggesting a high probability that points with even smaller dissimilarity exist outside the local search range.

[0090] Alternatively, a threshold Th1 may be set, and the parallax invalidity determination unit 103 may determine whether the following equation (2) is true and turn on this function when equation (2) is true.

[0091] [Math 2] SADmax-LRmin ≧ Th1 (2)

[0092] This approach is adopted because large fluctuations in dissimilarity within the local search range suggest the existence of points with smaller dissimilarity around the local search range.

[0093] Similarly, a threshold Th2 may be set, and the parallax invalidity determination unit 103 may determine whether the following equation (3) is true and turn on this function when equation (3) is true.

[0094] [Math 3] SADmax-SADmin ≧ Th2 (3)

[0095] In this configuration, the disparity invalidity determination unit 103 in the arithmetic unit 1 can be configured to determine whether or not to expand the first search width (higher-level search width) or the second search range (lower-level search range) to perform a search based on whether or not the luminance gradient (dissimilarity gradient; for example, SADmin-LRmin, SADmax-LRmin, or SADmax-SADmin) within the image block is greater than a predetermined threshold (for example, Th0, Th1, or Th2). According to this embodiment with such a configuration, disparity searching can be performed efficiently according to the luminance gradient within the image block.

[0096] In addition, in the computing device 1, the parallax invalidity determination unit 103 may be configured to decide whether or not to expand the first search width or the second search range and perform the search based on whether or not the first parallax obtained in the first-level stereo matching process (local search of the higher level) is greater than a predetermined threshold. According to this embodiment with such a configuration, parallax search can be performed efficiently according to the proximity of the distance from the two cameras to the object (relationship between the first parallax and the threshold).

[0097] <Third Embodiment> Next, an example of an arithmetic device (image processing device) according to the third embodiment will be described with reference to Figure 10. Figure 10 is a flowchart showing an example of the procedure for local search processing of the Nth hierarchical search by the disparity calculation unit 100A of the third embodiment. The disparity calculation unit 100A in this embodiment can use the same configuration as the disparity calculation unit 100A in the second embodiment. The processing in steps S21 to S25 in Figure 10 is the same as the processing in steps S11 to S15 in Figure 8 of the second embodiment, so the explanation is omitted.

[0098] First, in the disparity calculation unit 100A, the higher-level reference disparity determination unit 102 and the disparity search unit 101 perform the processes of steps S21 to S25.

[0099] After processing in step S25, the parallax invalidity determination unit 103 determines, based on the search results from the parallax search unit 101, whether or not there are multiple local minimums of dissimilarity within the local search range (step S26).

[0100] In step S26, if there is only one local minimum value of dissimilarity within the local search range (NO in step S26), the parallax invalidity determination unit 103 determines whether it is valid or invalid according to the location of the local minimum value within the local search range (whether the distance from the edge is closer than the threshold), and outputs the parallax and whether the parallax is valid or invalid (step S28).

[0101] On the other hand, if multiple local minima of dissimilarity exist within the local search range (YES in step S26), the parallax invalidity determination unit 103 determines that the parallax is invalid and outputs the determination result (step S27). Information on parallax and parallax valid / invalid is output to the vehicle control device 4 and the higher-level reference parallax determination unit 102. After processing in step S27 or S28, the local search process of the Nth hierarchical search is terminated.

[0102] As described above, the image processing device (arithmetic unit 1) according to the third embodiment can be configured such that, in the first-level stereo matching process (local search of the upper level), if there are multiple image blocks that are considered similar to the first image block, the first disparity is not output as a result of the first-level stereo matching process, and the second-level stereo matching process (local search of the lower level) is performed.

[0103] According to this embodiment with the above configuration, in the first-level stereo matching process, if there are multiple image blocks that are considered similar to the first image block, the calculated disparity is deemed invalid, and the second-level stereo matching process (local search of lower levels) is performed. This avoids invalidating all corresponding lower-level disparities, thereby increasing the likelihood of calculating disparity.

[0104] Furthermore, in the arithmetic unit 1, the parallax invalidity determination unit 103 can be configured to determine whether or not to output a first parallax based on whether or not the luminance gradient within the image block is greater than a predetermined threshold, by utilizing the configuration of the second embodiment. According to this embodiment with such a configuration, parallax search can be performed efficiently according to the luminance gradient within the image block.

[0105] <Variation> Furthermore, the present invention is not limited to the embodiments described above, and of course, various other applications and modifications can be taken as long as they do not depart from the gist of the invention as described in the claims. For example, the embodiments described above are described in detail and specifically in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the components described. Also, it is possible to replace a part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, replace, or delete other components for a part of the configuration of each embodiment.

[0106] Furthermore, in this specification, processing steps describing chronological processing include not only processing performed chronologically in the order described, but also processing that is not necessarily performed chronologically but is performed in parallel or individually (for example, processing by objects).

[0107] Furthermore, in the embodiments described above, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In practice, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0108] 1...Calculation unit, 2...Left camera, 3...Right camera, 4...Vehicle control device, 10...Vehicle, 20...Subject, 41...Distance image, 45...Coarse distance image, 46...Detailed distance image, 21...First camera image (left camera image), 31...Second camera image (right camera image), 100,100A...Disparity calculation unit, 101...Disparity search unit, 102...Higher reference disparity determination unit, 103...Disparity invalidity determination unit, 201,203...Image area, 202...Epipolar line, 501~503,901...Disparity

Claims

1. An image processing device including a parallax calculation unit that calculates parallax by comparing two images captured by two cameras, The parallax calculation unit a first-level stereo matching process for determining a first disparity for each image region of the first size by identifying an image block that is an image region of a first size and has the highest similarity to a first image block included in one of the two images from a first search range determined based on a predetermined first search width in the other of the two images; performing a second-level stereo matching process in which a second disparity is determined for each image region of the second size by identifying an image block in the other image of the two images, the image block being an image region of a second size smaller than the first size and having the highest similarity to a second image block included in one of the two images, from a second search range determined based on the first disparity and a predetermined second search width; In the first-layer stereo matching process, when an image block having the highest similarity to the first image block exists at an edge of the first search range, the first search width or the second search range is widened and a search is performed. Image processing device.

2. When the first search width or the second search range is widened to perform a search, an upper limit is set in advance for the first search width or the second search range. The image processing device according to claim 1 .

3. the disparity calculation unit, when performing a search by widening the first search width or the second search range, terminates the search based on a point at which the similarity decreases along the search direction; The image processing device according to claim 1 .

4. and when an image block having the highest similarity to the first image block is present at an end of the first search range in the first-layer stereo matching process, the disparity calculation unit determines the first search width or the second search range based on at which end of the first search range the image block having the highest similarity is found. The image processing device according to claim 1 .

5. the disparity calculation unit determines whether to widen the first search width or the second search range and perform a search based on the size of the first image block. The image processing device according to claim 1 .

6. the disparity calculation unit determines whether to expand the first search range or the second search range and perform a search based on whether a difference in the similarity within the first search range is greater than a predetermined threshold in the search in the first layer stereo matching process. The image processing device according to claim 1 .

7. the disparity calculation unit determines whether to expand the first search width or the second search range and perform a search based on whether a difference between the similarity of the image block with the highest similarity and the similarities of other image blocks within the search range is greater than a predetermined threshold in the search in the first-layer stereo matching process. The image processing device according to claim 1 .

8. the disparity calculation unit determines whether to expand the first search width or the second search range and perform a search based on whether the first disparity calculated in the first-layer stereo matching process is greater than a predetermined threshold. The image processing device according to claim 1 .

9. the disparity calculation unit determines whether to expand the first search width or the second search range and perform a search based on whether a luminance gradient in the image block is greater than a predetermined threshold. The image processing device according to claim 1 .

10. the disparity calculation unit, when there are a plurality of image blocks that are considered to be similar to the first image block in the first-layer stereo matching process, does not output the first disparity as a result of the first-layer stereo matching process, and performs the second-layer stereo matching process; The image processing device according to claim 1 .

11. the disparity calculation unit determines whether to output the first disparity based on whether a luminance gradient in the image block is greater than a predetermined threshold. The image processing device according to claim 10.

12. An image processing method using an image processing device having a parallax calculation unit that calculates parallax by comparing two images captured by two cameras, The parallax calculation unit a first-level stereo matching process for determining a first disparity for each image region of the first size by identifying an image block that is an image region of a first size and has the highest similarity to a first image block included in one of the two images from a first search range determined based on a predetermined first search width in the other of the two images; performing a second-level stereo matching process in which a second disparity is determined for each image region of the second size by identifying an image block in the other image of the two images, the image block being an image region of a second size smaller than the first size and having the highest similarity to a second image block included in one of the two images, from a second search range determined based on the first disparity and a predetermined second search width; In the first-layer stereo matching process, when an image block having the highest similarity to the first image block exists at an edge of the first search range, the first search width or the second search range is widened and a search is performed. Image processing methods.

Images