Image processing device and image processing method

Abstract

To increase a possibility of parallax calculation even when invalidity of a higher-order parallax is determined in hierarchical search.SOLUTION: An image processing device for determining parallax by comparing images captured by two cameras includes: first hierarchical matching processing for obtaining first parallax in image region units of a first size by identifying an image block which has a high degree of similarity to a first image block contained in one of the two images from a first search range determined on the basis of a predetermined first search width in the other image; and second hierarchical matching processing for obtaining second parallax in image region units of a second size by identifying the same from a second search range determined on the basis of the first parallax and a predetermined second search width in the other image. The image processing device performs a search by broadening the first search width or the second search range when an image block which has a high degree of similarity to the first image block exists, at an end within the first search range during the first hierarchical matching processing.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method for monitoring objects present around a vehicle using images captured by an imaging device. [Background technology]

[0002] There is a known position measurement device that uses the principle of triangulation to measure the distance to an object within the field of view of a pair of stereo cameras, based on stereo images taken by the pair of cameras. The principle of triangulation is to calculate the distance from the camera to the object using the positional difference (parallax) between the images of the same object taken by the left and right cameras. The parallax is derived by identifying where the image of the object in one image is located in the other image.

[0003] Various methods have been proposed to derive disparity. For example, a classical method is block matching, which searches for an image region consisting of multiple pixels in one image and an image region in the other image that has the lowest dissimilarity.

[0004] In recent years, such distance measurement results have been used in autonomous driving (AD) and advanced driver assistance systems (ADAS). In autonomous driving, it is essential to detect objects that have fallen on the road at a greater distance and to understand road surface conditions such as holes and bumps. However, since distant objects are composed of fewer pixels in an image than close objects, more precise distance information is required. This poses the problem of increased computational complexity and time required to derive disparity.

[0005] As a countermeasure against the increase in the amount of calculation and calculation time, a hierarchical search is known in which a low-resolution disparity image is first created, and then, when searching for a higher-resolution disparity, only the corresponding low-resolution disparity, i.e., the vicinity of the higher-resolution disparity, is searched (see Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-319229 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, in order to calculate a disparity at a higher resolution (lower), only the vicinity of a pre-calculated low-resolution (higher) disparity is searched, thereby reducing the amount of calculation and the calculation time. However, when searching for a disparity, for example, the dissimilarity of the image regions of the left and right images becomes minimum at the edge of the disparity search range, and it is predicted that the true disparity exists outside the search range. Therefore, the reliability of the calculated disparity may be deemed low, and the disparity may be invalidated. In the hierarchical search, if a disparity at a lower resolution is deemed invalid, the high-resolution disparity corresponding to this low-resolution disparity cannot be searched and all high-resolution disparities are invalidated. This poses a problem in that a large amount of invalid disparity may occur in the final high-resolution disparity image.

[0008] In view of the above situation, there has been a demand for a method for increasing the possibility of calculating disparities by avoiding invalidating all of the corresponding lower disparities when a higher disparity is determined to be invalid in hierarchical search. [Means for solving the problem]

[0009] In order to solve the above problem, an image processing device according to one aspect of the present invention is an image processing device including a parallax calculation unit that calculates parallax by comparing two images captured by two cameras. The disparity calculation unit performs a first-layer stereo matching process to determine a first disparity for each image area of ​​the first size by identifying an image block that is an image area of ​​a first size and is included in one of the two images from a first search range determined based on a predetermined first search width in the other of the two images, and a second-layer stereo matching process to determine a second disparity for each image area of ​​the second size by identifying an image block that is an image area of ​​a second size smaller than the first size and is included in one of the two images from a second search range determined based on the first disparity and a predetermined second search width in the other of the two images. Then, in the first-layer stereo matching process, if the image block that is most similar to the first image block is located at an edge of the first search range, the disparity calculation unit widens the first search range or the second search range and performs a search. Here, the search range is information on the start and end points of the search that determines the search width. [Effects of the Invention]

[0010] According to at least one aspect of the present invention, when a higher-level disparity is determined to be invalid in a hierarchical search, it is possible to avoid invalidating all of the corresponding lower-level disparities, thereby increasing the possibility of disparity calculation. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a control system in a vehicle. [Figure 2] 1A and 1B are diagrams illustrating a comparison of the vehicle's surroundings, a rough range image, and a detailed range image. [Figure 3] 1 is a block diagram showing an example of the functional configuration of a calculation device according to a first embodiment of the present invention. [Figure 4] FIG. 3 is a diagram illustrating a concept of determining a disparity by a disparity searcher included in a disparity calculation unit according to the first embodiment of the present invention. [Figure 5] 5A and 5B are diagrams illustrating examples of determinations in local re-search and examples of search range extension directions during re-search according to the first embodiment of the present invention. [Figure 6] 10 is a flowchart showing an example of the procedure of local search processing in the Nth hierarchical layer according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a block diagram illustrating an example of the functional configuration of a disparity calculation unit in a calculation device according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart showing an example of the procedure of local search processing in the Nth hierarchical layer according to the second embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of magnitude of dissimilarity within a local search range in the second embodiment of the present invention. [Figure 10] 13 is a flowchart showing an example of the procedure of local search processing in the Nth hierarchical layer according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, examples of modes for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, identical components or components having substantially the same functions will be assigned the same reference numerals, and redundant explanations will be omitted. Furthermore, when there are multiple components having the same or similar functions, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0013] First Embodiment First, a calculation device (an example of an image processing device) according to a first embodiment of the present invention will be described with reference to FIGS.

[0014] FIG. 1 is a diagram showing an example of the hardware configuration of a control system in a vehicle. The vehicle 10 shown in FIG. 1 includes a calculation device 1, a left camera 2, a right camera 3, and a vehicle control device 4. The calculation device 1 and the vehicle control device 4 are electronic control units (ECUs). The left camera 2 and the right camera 3 output captured images obtained by capturing images of the surroundings of the vehicle 10 to the calculation device 1. The left camera 2 and the right camera 3 are arranged side by side in the horizontal direction. Hereinafter, the image captured by the left camera 2 will be referred to as a first camera image 21, and the image captured by the right camera 3 will be referred to as a second camera image 31. The left camera 2 will also be referred to as the "first camera," and the right camera 3 will also be referred to as the "second camera."

[0015] Because the left camera 2 and right camera 3 are positioned so as to face horizontally, both the first camera image 21 (left image) and the second camera image 31 (right image) are images obtained by capturing a three-dimensional space that extends in both the vertical and horizontal directions. However, the terms horizontal and vertical here do not have a strict meaning, but merely mean that the left and right directions of the first camera image 21 and the second camera image 31 correspond to the horizontal direction, not the top and bottom. For example, the left camera 2 and right camera 3 may be tilted by up to about 30 degrees around the optical axis as the center of rotation.

[0016] The computing device 1 includes a CPU 11, which is a central processing unit, a ROM 12, which is a read-only storage device, a RAM 13, which is a read / write storage device, and an interface 14 that receives a first camera image 21 and a second camera image 31. The CPU 11 loads a computer program stored in the ROM 12 into the RAM 13 and executes it, thereby realizing multiple functions of the computing device 1, which will be described later. The computer program may be stored in a non-volatile storage device (not shown) instead of the ROM 12. The ROM 12 and the non-volatile storage device are used as examples of computer-readable, non-transitory recording media that store the program.

[0017] However, instead of the combination of the CPU 11, the ROM 12, and the RAM 13, the arithmetic device 1 may implement some or all of the functions thereof using a field programmable gate array (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an integrated circuit for a specific application. Also, instead of the combination of the CPU 11, the ROM 12, and the RAM 13, the arithmetic device 1 may be implemented using a combination of different configurations, for example, a combination of the CPU 11, the ROM 12, the RAM 13, and an FPGA.

[0018] Interface 14 receives first camera image 21 (left image) and second camera image 31 (right image) and provides them to CPU 11. However, because the registers provided in CPU 11 can only store a small amount of information, interface 14 stores the received first camera image 21 and second camera image 31 in RAM 13 so that CPU 11 can freely refer to that information.

[0019] The arithmetic unit 1 uses the input first camera image 21 (left image) and second camera image 31 (right image) to create a distance image 41 (see FIG. 2) containing distance information to objects around the vehicle 10 based on parallax, and outputs the image to the vehicle control device 4. The process by which the arithmetic unit 1 creates the distance image 41 will be described in detail later with reference to FIG. 2 and subsequent figures. The distance image 41 represents the perspective of distance on a two-dimensional plane using color shading and color saturation. However, in this embodiment, each pixel of the distance image 41 stores an integer value between "0" and "255" that represents the shading. Among these, "0" is a special value indicating that distance calculation was not possible.

[0020] The range shown in distance image 41 is approximately the same as that of first camera image 21 (left image) and second camera image 31 (right image). However, the range shown in distance image 41 may be narrower than that of first camera image 21 and second camera image 31. The number of pixels in distance image 41 may be approximately the same as that of first camera image 21 and second camera image 31, or may be smaller than that of first camera image 21 and second camera image 31.

[0021] That is, it is sufficient if the three-dimensional space recorded in the first camera image 21 (left image) and the second camera image 31 (right image) overlap at least partially. The mounting positions and mounting orientations of the left camera 2 and the right camera 3 on the vehicle 10 are known, and the relative positional relationship between the first camera image 21 and the second camera image 31 is also known. Therefore, it is also known which position in the second camera image 31 the position of a predetermined pixel in the first camera image 21 corresponds to in the three-dimensional space. The relative positional relationship between the first camera image 21 and the second camera image 31 may be determined by calculation, or may be determined by referring to a lookup table or the like that is prepared in advance.

[0022] That is, the relative positions of the left camera 2 and the right camera 3 are known, and the internal parameters of the two cameras are also known, so the relationship between the distance of the subject to the camera and the parallax is also known. In this embodiment, the relationship between distance and parallax is stored in the ROM 12 of the computing device 1 as a parallax table (not shown). In this embodiment, since the relationship between distance and parallax is known, the parallax can also be called "information corresponding to distance." Furthermore, in this embodiment, distance and parallax can be easily converted into each other, so the two can be considered to be substantially the same information.

[0023] The vehicle control device 4 controls the vehicle 10 using the distance image 41 input from the calculation device 1. For example, the vehicle control device 4 detects obstacles present around the vehicle 10 from the distance image 41, and controls the vehicle 10 to avoid collision between the vehicle 10 and the detected obstacle. The control of the vehicle 10 by the vehicle control device 4 is realized, for example, by the vehicle control device 4 outputting operation commands to a steering device (not shown) and a braking device (not shown).

[0024] [View of the vehicle's surroundings, rough range image, and detailed range image] FIG. 2 shows a comparison of the surroundings of the subject, a rough range image, and a detailed range image. 2 is a diagram illustrating a comparison of the resolution of the information stored in first camera image 21, second camera image 31, rough distance image 45, and detailed distance image 46. Note that resolution here does not refer to the distance in the depth direction, but rather to the level of resolution in the vertical and horizontal directions on the image. Note that rough distance image 45 and detailed distance image 46 are examples of distance image 41 shown in FIG. 1.

[0025] The top part of Figure 2 shows the surroundings of vehicle 10, including the presence of another vehicle 20. The middle part of Figure 2 is a conceptual diagram showing the resolution of a rough distance image 45 captured of vehicle 20, and the bottom part of Figure 2 is a conceptual diagram showing the resolution of a detailed distance image 46, which has a higher resolution than rough distance image 45.

[0026] Rough distance image 45 shown in the middle of FIG. 2 stores distance information for a total of 25 image regions obtained by dividing the region shown in the top of FIG. 2 into five regions each vertically and horizontally.

[0027] Detailed distance image 46 shown in the lower part of Fig. 2 stores distance information for a total of 100 image regions, which are obtained by dividing the region shown in the upper part of Fig. 2 into 10 vertical and 10 horizontal regions. In rough distance image 45 and detailed distance image 46, the top and bottom of the illustrated images correspond to the vertical direction, and the left and right of the illustrated images correspond to the horizontal direction.

[0028] In this embodiment, the image area indicated by one piece of distance information included in rough distance image 45, as shown in the middle of Figure 2, is called a "first image block," and the image area indicated by one piece of distance information included in detailed distance image 46, as shown in the bottom of Figure 2, is called a "second image block." The size of the first image block (first size) is larger than the size of the second image block (second size). In the example shown in Figure 2 and this embodiment, the first image block is twice as wide and twice as high as the second image block, and its area is four times larger. However, as long as the first image block is larger than the second image block, there are no particular restrictions on the multiple.

[0029] Although the distance information for one point is not necessarily obtained by measuring the entire first image block, in this embodiment, the distance information is considered to represent the entire first image block for convenience. In addition, it is also known which first image block in rough distance image 45 each second image block in detailed distance image 46 falls within.

[0030] Each of the first camera image 21 and the second camera image 31 has brightness and saturation information at a resolution equal to or greater than that of the second image block. In accordance with the example of FIG. 2, each of the first camera image 21 and the second camera image 31 has information on at least 10 pixels in both the vertical and horizontal directions within the range shown in the upper part of FIG. 2, i.e., a total of 100 pixels or more. The first camera image 21 and the second camera image 31 may have approximately the same number of pixels as the detailed distance image 46, or may have twice or ten times as many pixels in the vertical and horizontal directions. In this embodiment, as an example, the first camera image 21 and the second camera image 31 have approximately the same number of pixels as the detailed distance image 46.

[0031] [Functional configuration of the disparity calculation unit] Next, an example of the functional configuration of the arithmetic device 1 according to the first embodiment of the present invention will be described with reference to FIG. Fig. 3 is a block diagram showing an example of a functional configuration of the arithmetic device 1. As shown in Fig. 3, the arithmetic device 1 includes a disparity calculation unit 100 including a disparity search unit 101, a higher-order reference disparity determination unit 102, and a disparity invalidity determination unit 103. The functions of the disparity search unit 101, the higher-order reference disparity determination unit 102, and the disparity invalidity determination unit 103 of the disparity calculation unit 100 are realized by software processing by the CPU 11 described above. Note that information generated by the processing described below, such as search range information, disparity information, dissimilarity information, higher-order disparity information, disparity invalidity information, and a search range change instruction, is temporarily stored in the RAM 13.

[0032] Furthermore, as described above, the principle of triangulation can be used to calculate the distance from the left and right cameras to the same subject based on the parallax between two left and right camera images that include the subject. Therefore, the parallax calculation unit 100 may generate distance images (e.g., a rough distance image 45, a detailed distance image 46, etc.) based on the parallax information of the two images and output them to the vehicle control device 4. However, in this embodiment, an example will be described in which the parallax calculation unit 100 calculates the parallax information and outputs it to the vehicle control device 4.

[0033] The parallax calculation unit 100 receives as input a first camera image 21 (left image) and a second camera image 31 (right image) captured by a left camera 21 and a right camera 31 from different viewpoints.

[0034] The disparity search unit 101 performs disparity search (disparity determination) by comparing two images, the first camera image 21 and the second camera image 31, and outputs "disparity information" and "dissimilarity information" used in the disparity determination as search results. For example, a stereo matching process can be used for this search process. In the stereo matching process, the image blocks to be compared in the two images are shifted by one pixel each to be compared. This stereo matching process is an example of a first-level stereo matching process. If the generated disparity is not the highest-level disparity in the hierarchical search (see, for example, Patent Document 1), the disparity search unit 101 obtains "search range information" from the higher-level reference disparity determination unit 102 and performs a local search that searches only within that range. This local search process is an example of a second-level stereo matching process. In the following description, disparity search will also be simply referred to as "search."

[0035] The above search range information is information on the start and end pixels that determine the search range (pixel shift amount) in another camera image (e.g., second camera image 31) relative to a reference camera image (e.g., first camera image 21) in parallax search using stereo matching processing. The search width is determined from the start and end points. Generally, the higher the resolution of the image, the wider the search width.

[0036] Furthermore, the highest disparity is the disparity found from the lowest resolution image when multiple hierarchical resolution images are generated from a camera image (original image). As a result of the search, the disparity search unit 101 determines the disparity that minimizes the dissimilarity. Information about the determined disparity (information representing the position of the disparity on the image) is sent to the disparity invalidity determination unit 103 together with dissimilarity information.

[0037] The parallax invalidity determination unit 103 determines whether the parallax determined by the parallax search unit 101 is located at the edge of the local search range by referring to the "dissimilarity information" output from the parallax search unit 101. If the determined position of the parallax is at the edge, the parallax invalidity determination unit 103 determines that there is a high possibility that a point with even lower dissimilarity, i.e., a true parallax, exists outside the local search range, and issues an instruction ("search range change instruction" in the drawing) to the parallax search unit 101 to change (e.g., extend) the local search range outside the edge of the local search range where the determined parallax is located and perform a search again.

[0038] Regarding the degree of extension of the local search range, the search range after extension and the extension width are determined in advance for each condition such as the image hierarchy (resolution) and are registered in the parallax calculation unit 100 (for example, in the ROM 12). The degree of extension of this local search range differs for each image hierarchy depending on, for example, the allowable search time and cost. For example, the search time differs depending on the processing capacity of the calculation device 1, the data size of the camera image, etc. The maximum extension width of the local search range is determined by the circuit scale (processing capacity) of the calculation device 1.

[0039] The disparity search unit 101 performs a search again in response to an instruction to change the search range from the disparity invalidity determination unit 103, and again outputs the point at which the dissimilarity is minimum, including the extended search range, as the disparity.

[0040] At this time, an upper limit may be set for the search range to be extended, taking into consideration the amount of calculation and calculation time of the calculation device 1. That is, when the calculation device 1 widens the first search range (search range in a higher hierarchy) or the second search range (search range in a lower hierarchy) to perform a search, an upper limit is set in advance for the first search range or the second search range. According to this embodiment configured in this manner, by setting an upper limit for the first search range or the second search range, it is possible to reduce the amount of calculation and shorten the calculation time of the calculation device 1.

[0041] Furthermore, when the disparity search unit 101 finds the first minimum value of dissimilarity during the re-search process, it may terminate the search at that point and output the point of the first minimum value as the disparity. That is, when the disparity search unit 101 performs a search by expanding the first search width (search width of the upper layer) or the second search range (search range of the lower layer), it may be configured to terminate the search based on the point at which the similarity decreases along the search direction. According to this embodiment configured in this manner, by determining the disparity based on the point at which the similarity decreases along the search direction, it is possible to reduce the amount of calculation by the calculation device 1 and shorten the calculation time.

[0042] In this specification, the terms "minimum" and "local minimum" are used, but in this specification, they include not only the minimum and local minimum in the strict mathematical sense but also the substantial minimum and local minimum. For example, a local minimum point corresponds to a point where the differential value is 0 or a predetermined value or less (for example, a small value close to 0).

[0043] Furthermore, whether or not to perform an extended search and the extension range may be changed depending on whether the extension direction of the local search range, which will be described later with reference to FIG. 5, is the same as or opposite to the search direction. That is, in the first-layer stereo matching process (local search of a higher layer), when an image block with the highest similarity to the first image block is present at an edge of the first search range, the disparity search unit 101 can be configured to determine the first search width or the second search range based on which edge of the first search range the most similar image block is found at. According to this embodiment configured in this manner, by determining the first search width or the second search range based on which edge of the first search range the image block with the highest similarity (the smallest dissimilarity) is found at, it is possible to search for a disparity with a higher similarity.

[0044] The "parallax information" and "dissimilarity information" resulting from the re-search are again sent from the parallax search unit 101 to the parallax invalidity determination unit 103. Here, the parallax invalidity determination unit 103 may send the results of the re-search performed once in the parallax search unit 101 as "parallax information" and "parallax invalidity (valid / invalid) information" to the vehicle control device 4 as they are, or may perform a determination for re-search again in the parallax invalidity determination unit 103. The basic idea of ​​validity / invalidity is that if the position of the determined disparity is not at the end of the local search range, the disparity is determined to be valid, and if the position of the determined disparity is at the end of the local search range, the disparity is determined to be invalid.

[0045] [Disparity determination by disparity search unit] Next, the concept of determining the disparity by the disparity search unit 101 (FIG. 3) included in the disparity calculation unit 100 according to the first embodiment of the present invention will be described with reference to FIG. Fig. 4 is a diagram showing the concept of disparity determination by disparity search unit 101 of disparity calculation unit 100. The upper part of Fig. 4 shows first camera image 21 and second camera image 31. In this example, a reference image corresponding to image area 201 of first camera image 21, which is the base image, i.e., the image area of ​​second camera image 31, is searched for. Note that while first camera image 21 is used as the base image, second camera image 31 may also be used as the base image.

[0046] In this embodiment, the left camera 2 and the right camera 3 are installed side by side in the horizontal direction, so the epipolar line 202 according to the epipolar geometry is shown as a dashed line extending in the horizontal direction. When the width in the illustrated horizontal direction of the first camera image 21 and the second camera image 31 is 1000 pixels and the center of the image area 201 is the 400th pixel from the left end of the first camera image 21, a numerical value of -400 to +600 is assigned in the horizontal direction of the second camera image 31, which is the reference image, as shown in the figure. For example, when the disparity search unit 101 calculates the distance 500 pixels from the left end of the reference image, eye If it is determined that the image area 203 centered on corresponds to the image area 201 of the reference image, the parallax is calculated as "100".

[0047] 4 is a diagram showing an example of the dissimilarity between the reference image (second camera image 31) and image region 201 at each position on epipolar line 202. In the example shown in the middle of Fig. 4, there is a peak (minimum point) where the dissimilarity is minimum, and there are no other peaks with similar values, so disparity search unit 101 calculates the disparity of image region 201 to be "100".

[0048] The lower part of FIG. 4 shows another example of the dissimilarity between the reference image (second camera image 31) and image region 201 at each position on epipolar line 202. In the example shown in the lower part of FIG. 4, there are two peaks where the dissimilarity is minimum, and the dissimilarity values ​​are approximately the same, so it is not possible to determine whether the disparity of image region 201 is "-180" or "+115." In such a case, disparity search unit 101 determines that image region 201 is an invalid region and sets a disparity of "0" at the position of distance image 41 corresponding to image region 201. Note that a related invention in which there are multiple peaks where the dissimilarity is minimum is shown in the third embodiment described later.

[0049] [Local re-search judgment example and search range extension direction during re-search] 5 is a diagram showing an example of a determination made in local re-search according to the first embodiment of the present invention and an example of the direction in which the search range is extended during re-search. The horizontal axis of the graph in the figure represents the position (corresponding to the pixel shift amount) when comparing first camera image 21 and second camera image 31, and the vertical axis represents the dissimilarity of the comparison result. The area between two dashed dotted lines indicates the local search range.

[0050] In the graph in the upper part of FIG. 5, the dissimilarity of disparities within the local search range monotonically decreases, and the dissimilarity is minimum at the right end of the local search range. Therefore, the disparity search unit 101 outputs the disparity 501 at the right end of the local search range, and simultaneously outputs the five dissimilarities calculated here (including the dissimilarity corresponding to the disparity 501). By acquiring the five dissimilarities along with the disparity 501 from the disparity search unit 101, the disparity invalidity determination unit 103 can determine that the disparity 501 is located at the right end of the local search range, i.e., that the disparity 501 is invalid. Therefore, the disparity 501 becomes a target for a search range change instruction from the disparity invalidity determination unit 103.

[0051] Alternatively, the disparity search unit 101 may output, instead of the dissimilarity, that the output disparity is at the end or the right end of the local search range. From this information, the disparity invalidity determination unit 103 can determine that the disparity output by the disparity search unit 101 is at the end (here, the right end) of the local search range. In the example shown in the upper part of FIG. 5, the extension direction of the search range during re-search is to the right (the direction in which the pixel shift amount increases).

[0052] 5, the dissimilarity of disparity within the local search range does not monotonically decrease but has a maximum value, but the disparity search unit 101 still outputs disparity 502 located at the right end of the local search range as the point of minimum dissimilarity. Therefore, disparity 502 is subject to a search range change instruction from the disparity invalidity determination unit 103. The search range extension direction during re-search is to the right (the direction in which the pixel shift amount increases).

[0053] 5, the dissimilarity of disparity within the local search range does not increase monotonically but has maximum and minimum values, with the dissimilarity of disparity 503 located at the left end of the local search range being the smallest. Therefore, disparity 503 is subject to a search range change instruction from the parallax invalidity determination unit 103. The search range extension direction during re-search is to the left (the direction in which the pixel shift amount decreases).

[0054] [Example of local search processing procedure] Next, an example of the procedure of local search processing in the Nth hierarchical layer in the first embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the procedure of local search processing in the Nth hierarchical layer by the disparity calculation unit 100 of the first embodiment.

[0055] First, in the disparity calculation unit 100, the higher-level reference disparity determination unit 102 refers to the disparity of the (N-1)th layer (lower resolution) higher than the Nth layer obtained from the disparity invalidity determination unit 103 (step S1), and determines an initial search range for the Nth layer (step S2). Note that if the Nth layer is the highest layer (lowest resolution), the search range is set in advance and stored in the ROM 12.

[0056] Next, the disparity search unit 101 sets the initial search range determined by the higher-order reference disparity determination unit 102 as the search range for the local search, and performs a local search in the search range (step S3).

[0057] Next, the parallax invalidity determining unit 103 determines whether the point with the minimum dissimilarity is located at the edge of the search range based on the search result by the parallax searching unit 101 (step S4).

[0058] If the point at which the dissimilarity is minimum in step S4 is not at the edge of the search range (NO in step S4), the parallax invalidity determination unit 103 determines and outputs the parallax and whether the parallax is valid or invalid (valid in this case) (step S7). The information on the parallax and whether the parallax is valid or invalid is output to the vehicle control device 4 and the higher-level reference parallax determination unit 102.

[0059] On the other hand, if the point at which the dissimilarity is minimum is at the edge of the search range (YES in step S4), the parallax invalidity determination unit 103 determines whether the number of re-searches is within a preset upper limit of the number of re-searches (step S5). If the number of re-searches is within the preset upper limit of the number of re-searches (YES in step S5), the parallax invalidity determination unit 103 issues a search range change instruction to the parallax search unit 101. Note that the determination process in step S5 is provided in consideration of the allowable processing time, but may be deleted.

[0060] Next, when the disparity search unit 101 receives an instruction to change the search range, it corrects the search range used in the local search in step S3 (step S6) and searches for disparity again (step S3).

[0061] Next, after the YES determination in step S4, if the parallax invalidity determination unit 103 determines that the number of re-searches has reached a preset upper limit of the number of re-searches (NO in step S5), it confirms and outputs the parallax and whether the parallax is valid or invalid (step S7). After the processing of step S7, the local search process of the Nth hierarchical search layer is terminated.

[0062] As described above, the image processing device (arithmetic device 1) according to the first embodiment is an image processing device equipped with a parallax calculation unit (parallax calculation unit 100) that calculates the parallax by comparing two images (first camera image 21, second camera image 31) captured by two cameras (left camera 2, right camera 3). The disparity calculation unit performs a first-level stereo matching process (higher-level local search) to determine a first disparity in units of image areas of a first size by identifying an image block (e.g., image area 203) that is an image area of ​​a first size and is most similar to a first image block (e.g., image area 201) included in one of the two images from a first search range (local search range) determined based on a predetermined first search width in the other of the two images, and a second-level stereo matching process (lower-level local search (re-search)) to determine a second disparity in units of image areas of a second size by identifying an image block that is an image area of ​​a second size smaller than the first size and is most similar to a second image block included in one of the two images from a second search range determined based on the first disparity and a predetermined second search width in the other of the two images. Then, in the first-layer stereo matching process, if an image block with the highest similarity to the first image block exists at the edge of the first search range, the disparity calculation unit widens the first search width or the second search range and performs a search (for example, based on the modified (extended) local search range).

[0063] According to the present embodiment having the above configuration, when a higher-level disparity is determined to be invalid in the hierarchical search, it is possible to avoid invalidating all of the corresponding lower-level disparities, thereby increasing the possibility of disparity calculation. Furthermore, since high-resolution distance information can be calculated without increasing the amount of calculation or calculation time required for deriving disparity, even distant objects can be grasped with high accuracy. Therefore, according to this embodiment, it is possible to detect objects that have fallen on the road at a greater distance and to grasp road surface conditions such as potholes and bumps. This embodiment is suitable for application to autonomous driving (AD) and advanced driver assistance systems (ADAS).

[0064] Furthermore, this embodiment is configured to change the local search range (or search width) and perform a search again even if the disparity is determined to be invalid. Therefore, this embodiment not only finds the true minimum (or local minimum) of the dissimilarity (or similarity) from two images, but also feeds back to the same or lower layer in the hierarchical search, making it possible to efficiently search for a valid disparity without increasing the calculation load even if the disparity is once determined to be invalid.

[0065] <Second embodiment> The second embodiment is an example configured to output an invalid determination for a layer when the disparity dissimilarity reaches a minimum value at the edge of the local search range. The second embodiment will be described below, focusing mainly on the differences from the first embodiment.

[0066] [Functional configuration of the disparity calculation unit] 7 is a block diagram showing an example of the functional configuration of a disparity calculation unit in a calculation device (an example of an image processing device) according to a second embodiment of the present invention. As shown in Fig. 7, the calculation device 1 includes a disparity calculation unit 100A including a disparity search unit 101, a higher-order reference disparity determination unit 102, and a disparity invalidity determination unit 103. However, the disparity calculation unit 100A of the calculation device 1 according to this embodiment differs from the disparity calculation unit 100 according to the first embodiment (Fig. 3) in that there is no signal line for issuing a search range change instruction from the higher-order reference disparity determination unit 102 to the disparity search unit 101.

[0067] As in the first embodiment, the disparity search unit 101 receives the first camera image 21 (left image) and the second camera image 31 (right image) captured by the left camera 21 and the right camera 31 from different viewpoints.

[0068] As in the first embodiment, the disparity search unit 101 performs disparity search (disparity determination) by comparing two images, the first camera image 21 and the second camera image 31, and outputs "disparity information" and "dissimilarity information" used in the disparity determination as search results. The search process here is an example of first-layer stereo matching processing. If the disparity generated at this time is not the highest disparity in the hierarchical search (see, for example, Patent Document 1), the disparity search unit 101 obtains "search range information" from the higher-level reference disparity determination unit 102 and performs a local search that searches only within that range. This local search process is an example of second-layer stereo matching processing.

[0069] As a result of the search, the disparity search unit 101 determines the disparity so that the dissimilarity is minimized. Information on the determined disparity (information indicating the position of the disparity on the image) is sent to the disparity invalidity determination unit 103 together with the dissimilarity information.

[0070] The parallax invalidity determination unit 103 determines whether the parallax determined by the parallax search unit 101 is located at the edge of the local search range, with reference to the "dissimilarity information" output from the parallax search unit 101. If the determined parallax position is at the edge, the parallax invalidity determination unit 103 determines that the parallax is invalid, assuming that there is a high possibility that a point with even lower dissimilarity, i.e., a true parallax, exists outside the local search range. The parallax invalidity determination unit 103 outputs the "parallax information" and "parallax invalidity (valid / invalid) information" output from the parallax search unit 101 to the vehicle control device 4 and the higher-level reference parallax determination unit 102. The parallax determined as invalid here is referred to as a higher-level parallax during local search of a lower layer, but if the invalid parallax remains, all lower-level parallaxes are determined to be invalid. In such cases, for example, by using the method described in JP 2020-140344 A, it may be possible to avoid invalid judgment and output the lower disparity by performing a local search around the search range of the adjacent higher disparity.

[0071] [Example of local search processing procedure] Next, an example of the procedure of local search processing in the Nth hierarchical layer in the second embodiment of the present invention will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the procedure of local search processing in the Nth hierarchical layer by the disparity calculation unit 100A of the second embodiment.

[0072] First, in the disparity calculation unit 100, the higher-level reference disparity determination unit 102 refers to the disparity of the (N-1)th layer (lower resolution) higher than the Nth layer obtained from the disparity invalidity determination unit 103 (step S11). Then, the higher-level reference disparity determination unit 102 determines an initial search range in the Nth layer.

[0073] Next, the higher-level reference parallax determination unit 102 determines whether the higher-level parallax is invalid (step S12), and if the higher-level parallax is not invalid (NO in step S12), the process proceeds to step S14.

[0074] If it is determined in step S12 that the higher parallax is invalid (YES in step S12), the higher reference parallax determination unit 102 switches the reference destination to a parallax in a higher layer adjacent to the higher parallax (step S13).

[0075] If the determination result in step S12 is NO or after the processing of step S13, the disparity searching unit 101 sets a search range for the local search. If the determination result in step S12 is NO, the disparity searching unit 101 sets the initial search range determined by the higher-level reference disparity determining unit 102 as the search range for the local search (step S14). Furthermore, after the processing of step S13, the disparity searching unit 101 sets the search range for the local search around the adjacent disparity in the higher layer referenced by the higher-level reference disparity determining unit 102 (step S14).

[0076] Next, the disparity search unit 101 performs a local search within the set search range of the local search (step S15).

[0077] Next, the parallax invalidity determining unit 103 determines whether the point with the minimum dissimilarity is located at the edge of the search range based on the search result by the parallax searching unit 101 (step S16).

[0078] If the point where the dissimilarity is smallest in step S16 is not at the edge of the search range (NO in step S16), the parallax invalidity determining unit 103 determines and outputs the parallax and parallax validity (step S18).

[0079] On the other hand, if the point with the smallest dissimilarity is at the edge of the search range (YES in step S16), the parallax invalidity determination unit 103 determines that the parallax is invalid and outputs the determination result (step S17). The parallax and information on whether the parallax is valid or invalid are output to the vehicle control device 4 and the higher-level reference parallax determination unit 102. After the processing of step S17 or S18, the local search process of the Nth hierarchical search layer is terminated.

[0080] In this embodiment, the process of determining whether the number of re-searches is within the upper limit in step S5 of FIG. 3 described in the first embodiment may be provided after the process of step S16.

[0081] Furthermore, the invalidity determination described in this embodiment does not need to be uniform for all local searches. For example, in this embodiment, in a relatively higher layer (e.g., the lowest-resolution layer of the three layers) where a wider range than an arbitrary threshold is searched in one parallax search due to low resolution, the function of changing the local search range when a parallax invalidity determination is made in this invention (hereinafter referred to as "this function") can be turned off, and conversely, in a relatively lower layer where a narrower range is searched in one search due to high resolution, this function can be turned on.

[0082] In this way, in the arithmetic device 1 (image processing device), the disparity search unit 101 can be configured to determine whether to perform a search by expanding the first search width (search width of the higher layer) or the second search range (search range of the lower layer) based on the size of the first image block (image block of the camera image of the higher layer). According to this embodiment configured as described above, disparity search can be performed efficiently depending on the size of the first image block included in one of the two images.

[0083] In addition, when the difference in dissimilarity within the local search is greater than an arbitrary threshold, it is assumed that there is a high probability that the minimum value exists outside the local search range, and this function is turned on. Conversely, when the difference in dissimilarity is small, it is assumed that there is a low probability that the minimum value exists outside the local search range, and this function is turned off.

[0084] In this way, in the arithmetic device 1 (image processing device), the disparity search unit 101 can be configured to determine whether to expand the first search width or the second search range and perform a search based on whether the difference in similarity within the first search range is greater than a predetermined threshold in the search in the first-layer stereo matching process (local search in a higher layer). According to this embodiment configured as described above, disparity search can be performed efficiently in accordance with the difference in similarity within the first search range.

[0085] It is also possible to turn on this function when the dissimilarity of the disparity near the end of the local search range that has the smallest value differs by a predetermined amount or more from the dissimilarity of other disparities.

[0086] In this way, in the arithmetic device 1 (image processing device), the disparity invalidity determination unit 103 can be configured to determine whether to expand the first search width or the second search range and perform a search based on whether the similarity of the most similar image block in the search in the first-layer stereo matching process (local search in the upper layer) is different from the similarities of other image blocks within the search range by a larger than predetermined threshold. According to this embodiment configured as described above, disparity search can be performed efficiently depending on the difference in similarity within the first search range.

[0087] [Example of local search processing procedure] Hereinafter, the on / off switching of this function will be described with reference to FIG. FIG. 9 is a diagram showing an example of magnitude of dissimilarity within a local search range in the second embodiment of the present invention. For example, as shown in Fig. 9, the smaller of the dissimilarities at the left and right ends of the local search range is set to "LRmin", the maximum dissimilarity excluding the left and right ends is set to "SADmax", and the minimum is set to "SADmin". Here, a threshold value Th0 is defined, and the disparity invalidity determination unit 103 may determine whether the following formula (1) is satisfied, and turn on this function when formula (1) is satisfied.

[0088] [Number 1] SADmin-LRmin ≧ Th0 (1)

[0089] The reason for this operation is that the dissimilarity drops significantly at the left and right ends of the local search range, and therefore there is a high possibility that points with smaller dissimilarity exist outside the local search range.

[0090] Alternatively, a threshold value Th1 may be determined, and the parallax invalidity determining unit 103 may determine whether the following formula (2) is satisfied, and turn on this function when formula (2) is satisfied.

[0091] [Number 2] SADmax-LRmin ≧ Th1 (2)

[0092] The reason for this operation is that it can be inferred that there are points with smaller dissimilarities around the local search range due to large fluctuations in dissimilarity within the local search range.

[0093] Similarly, a threshold value Th2 may be determined, and the parallax invalidity determining unit 103 may determine whether the following formula (3) is satisfied, and turn on this function when formula (3) is satisfied.

[0094] [Number 3] SADmax-SADmin ≧ Th2 (3)

[0095] In this way, in the computing device 1, the parallax invalidity determination unit 103 can be configured to determine whether to expand the first search width (search width of a higher layer) or the second search range (search range of a lower layer) and perform a search, based on whether the luminance gradient (gradient of dissimilarity; for example, SADmin-LRmin, SADmax-LRmin, or SADmax-SADmin) in the image block is greater than a predetermined threshold (for example, Th0, Th1, or Th2). According to this embodiment configured as described above, it is possible to perform an efficient parallax search in accordance with the luminance gradient in the image block.

[0096] In the computing device 1, the parallax invalidity determination unit 103 may be configured to determine whether to expand the first search width or the second search range and perform a search, based on whether the first parallax calculated in the first-layer stereo matching process (local search in a higher layer) is greater than a predetermined threshold. According to this embodiment configured as described above, it is possible to perform an efficient parallax search depending on the distances from the two cameras to the object (the relationship between the first parallax and the threshold).

[0097] <Third embodiment> Next, a calculation device (an example of an image processing device) according to a third embodiment will be described with reference to FIG. Fig. 10 is a flowchart showing an example of the procedure of local search processing of the Nth hierarchical search layer by the disparity calculation unit 100A of the third embodiment. The disparity calculation unit 100A of this embodiment can use the same configuration as the disparity calculation unit 100A of the second embodiment. The processing of steps S21 to S25 in Fig. 10 is the same as the processing of steps S11 to S15 in Fig. 8 of the second embodiment, and therefore description thereof will be omitted.

[0098] First, in the disparity calculation unit 100A, the higher-level reference disparity determination unit 102 and the disparity search unit 101 perform the processes of steps S21 to S25.

[0099] After the process of step S25, the parallax invalidity determining unit 103 determines whether or not there are multiple minimum values ​​of dissimilarity within the local search range, based on the search result by the parallax searching unit 101 (step S26).

[0100] If there is one minimum value of dissimilarity within the local search range in step S26 (NO in step S26), the parallax invalidity determination unit 103 determines whether the disparity is valid or invalid depending on the position of the minimum value within the local search range (whether the distance to the edge is closer than a threshold), and determines and outputs the parallax and whether the parallax is valid or invalid (step S28).

[0101] On the other hand, if multiple minimum values ​​of dissimilarity exist within the local search range (YES in step S26), the parallax invalidity determination unit 103 determines that the parallax is invalid and outputs the determination result (step S27). The parallax and information on whether the parallax is valid or invalid are output to the vehicle control device 4 and the higher-level reference parallax determination unit 102. After the processing of step S27 or S28, the local search process of the Nth hierarchical search layer is terminated.

[0102] As described above, the image processing device (arithmetic device 1) according to the third embodiment can be configured to perform a second-layer stereo matching process (local search at a lower layer) without outputting the first disparity as a result of the first-layer stereo matching process when there are multiple image blocks that are considered to be similar to the first image block in the first-layer stereo matching process (local search at a higher layer).

[0103] According to this embodiment having the above configuration, in the first-layer stereo matching process, if there are multiple image blocks that are considered to be similar to the first image block, the calculated disparity is determined to be invalid, and the second-layer stereo matching process (local search of the lower layer) is performed, thereby avoiding invalidating all corresponding lower-layer disparities and increasing the possibility of disparity calculation.

[0104] Furthermore, in the arithmetic device 1, the parallax invalidity determination unit 103 can be configured to use the configuration of the second embodiment and determine whether to output the first parallax based on whether the luminance gradient in the image block is greater than a predetermined threshold. According to this embodiment configured as described above, it is possible to efficiently perform parallax search according to the luminance gradient in the image block.

[0105] <Modification> Furthermore, the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments have been described in detail and specifically to clearly explain the present invention, and are not necessarily limited to those including all of the components described. Furthermore, it is possible to replace part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components from part of the configuration of each embodiment.

[0106] In addition, in this specification, processing steps describing chronological processing include not only processing that is performed chronologically in the order described, but also processing that is not necessarily performed chronologically but is performed in parallel or individually (for example, processing by objects).

[0107] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0108] 1...Calculation device, 2...Left camera, 3...Right camera, 4...Vehicle control device, 10...Vehicle, 20...Subject, 41...Distance image, 45...Rough distance image, 46...Detailed distance image, 21...First camera image (left camera image), 31...Second camera image (right camera image), 100, 100A...Disparity calculation unit, 101...Disparity search unit, 102...Higher reference disparity determination unit, 103...Disparity invalidity determination unit, 201, 203...Image area, 202...Epipolar line, 501-503, 901...Disparity

Claims

1. An image processing device including a parallax calculation unit that calculates parallax by comparing two images captured by two cameras, The parallax calculation unit a first-level stereo matching process for determining a first disparity in units of image regions of the first size by identifying an image block that is an image region of a first size and has the highest similarity to a first image block included in one of the two images from a first search range that has information on pixels of a start point and an end point according to a pixel shift amount and that is determined based on a predetermined first search width in the other of the two images; performing a second-layer stereo matching process to determine a second disparity in units of image regions of the second size by identifying an image block that is an image region of a second size smaller than the first size and that has the highest similarity to a second image block included in one of the two images from a second search range that is determined based on a predetermined second search width in the other of the two images and has information on pixels of start and end points corresponding to the first disparity; In the first-layer stereo matching process, when an image block having the highest similarity to the first image block exists at an edge of the first search range, the first search width in the first-layer stereo matching process or the second search range in the second-layer stereo matching process is widened and a search is performed. Image processing device.

2. When the first search width or the second search range is widened to perform a search, an upper limit is set in advance for the first search width or the second search range. The image processing device according to claim 1 .

3. the disparity calculation unit, when performing a search by widening the first search width or the second search range, terminates the search based on a point at which the similarity decreases along the search direction; The image processing device according to claim 1 .

4. and when an image block having the highest similarity to the first image block is present at an end of the first search range in the first-layer stereo matching process, the disparity calculation unit determines the first search width or the second search range based on at which end of the first search range the image block having the highest similarity is found. The image processing device according to claim 1 .

5. the disparity calculation unit determines whether to widen the first search width or the second search range and perform a search based on the size of the first image block. The image processing device according to claim 1 .

6. the disparity calculation unit determines whether to expand the first search range or the second search range and perform a search based on whether a difference in the similarity within the first search range is greater than a predetermined threshold in the search in the first layer stereo matching process. The image processing device according to claim 1 .

7. the disparity calculation unit determines whether to expand the first search width or the second search range and perform a search based on whether a difference between the similarity of the image block with the highest similarity and the similarities of other image blocks within the search range is greater than a predetermined threshold in the search in the first-layer stereo matching process. The image processing device according to claim 1 .

8. the disparity calculation unit determines whether to expand the first search width or the second search range and perform a search based on whether the first disparity calculated in the first-layer stereo matching process is greater than a predetermined threshold. The image processing device according to claim 1 .

9. the disparity calculation unit determines whether to expand the first search width or the second search range and perform a search based on whether a luminance gradient in the image block is greater than a predetermined threshold. The image processing device according to claim 1 .

10. the disparity calculation unit, when there are a plurality of image blocks that are considered to be similar to the first image block in the first-layer stereo matching process, does not output the first disparity as a result of the first-layer stereo matching process, and performs the second-layer stereo matching process; The image processing device according to claim 1 .

11. the disparity calculation unit determines whether to output the first disparity based on whether a luminance gradient in the image block is greater than a predetermined threshold. The image processing device according to claim 10.

12. An image processing method using an image processing device having a parallax calculation unit that calculates parallax by comparing two images captured by two cameras, The parallax calculation unit a first-level stereo matching process for determining a first disparity in units of image regions of the first size by identifying an image block that is an image region of a first size and has the highest similarity to a first image block included in one of the two images from a first search range that has information on pixels of a start point and an end point according to a pixel shift amount and that is determined based on a predetermined first search width in the other of the two images; performing a second-layer stereo matching process to determine a second disparity in units of image regions of the second size by identifying an image block that is an image region of a second size smaller than the first size and that has the highest similarity to a second image block included in one of the two images from a second search range that is determined based on a predetermined second search width in the other of the two images and has information on pixels of start and end points corresponding to the first disparity; In the first-layer stereo matching process, when an image block having the highest similarity to the first image block exists at an edge of the first search range, the first search width in the first-layer stereo matching process or the second search range in the second-layer stereo matching process is widened and a search is performed. Image processing methods.

Images