System and method for generating a disparity map by comparing stereo images

Abstract

System (100, 200) for generating a disparity map, the system comprising: an image acquisition element (110, 210) that acquires a left image and a right image; an adjustment cost calculator (120, 220) that calculates adjustment costs for each of a plurality of pixels of the left image and the right image, a cumulative and additive calculator (130, 230) that calculates a cumulative value of one of the pixels based on the calculated adjustment costs and calculates a relaxation cumulative value that is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of one pixel and disparity values ​​of pixels surrounding that one pixel by the cumulative value of that one pixel, a disparity value derivation element (140, 240) that derives a disparity value for each of the pixels based on the calculated relaxation accumulation value, and a disparity map generator (150, 250) that generates the disparity map based on the derived disparity value, wherein, when the cumulative and additive calculator (130, 230) cumulatively and additively calculates the adjustment costs in a vertical direction, the cumulative and additive calculator (130, 230) calculates a current cumulative value by adding a minimum value of a cumulative value of a previous direction around the corresponding disparity, a value obtained by adding a first cumulative constant to a cumulative value of a direction around a disparity value, and a value obtained by adding a second cumulative constant to a minimum value of the cumulative values ​​of the previous direction, to current adjustment costs, and When the cumulative and additive calculator (130, 230) cumulatively and additively calculates the adjustment costs in a horizontal direction, the cumulative and additive calculator (130, 230) calculates the current cumulative value by adding a minimum value of the cumulative value of the previous direction of the corresponding disparity and the value obtained by adding the second cumulative constant to the minimum value of the cumulative values ​​of the previous direction, to the current adjustment costs.

Description

Background of the invention(a) Technical field

[0001] The present disclosure relates to a system and a method for generating a disparity map by matching stereo images. (b) Description of the related technique

[0002] Stereo image matching technology is widely used to detect situations around a vehicle (e.g., a motor vehicle), including object detection (e.g., a vehicle, a pedestrian, etc.) based on three-dimensional (3D) information, calculating a drivable area, and the like. To apply stereo image matching technology to a vehicle environment, high reliability of the stereo image matching results is required.

[0003] Since a global stereo image alignment algorithm can calculate high-quality (e.g., high-resolution) 3D information but is computationally complex, a local stereo image alignment algorithm, which requires less computational effort, is applied to the vehicle environment. However, the local alignment method has the disadvantage that it cannot perform an accurate and dense alignment over a wide area, such as a road surface.

[0004] Recently, studies have been conducted based on a semi-global alignment method, which offers high quality relative to its computational cost. However, the conventionally applied semi-global alignment method requires a 3D geometry element that is not suitable for the vehicle environment. This leads to a degradation in both alignment quality and computational effort. In particular, it is difficult to align a surface that is significantly inclined relative to a camera's image plane, such as a road surface, and it is difficult to perform stereo matching correctly under adverse weather conditions, such as rain or snow, which often occur during vehicle operation.

[0005] Therefore, it is necessary to develop a system and a procedure capable of generating an excellent disparity map by comparing stereo images, even under bad weather conditions.

[0006] Furthermore, the publication “Real-time disparity estimation algorithm for stereo camera systems” by Sang Hwa Lee and Siddharth Sharma in IEEE Transaction on Consumer Electronics, 2011, Vol. 57, No. 3, pages 1018-1026 discloses a system for generating a disparity map, the system comprising: an image acquisition element that acquires a left image and a right image, an adjustment cost calculator that calculates adjustment costs for each of a plurality of pixels of the left image and the right image, a cumulative and additive calculator that calculates a cumulative value of one of the pixels based on the calculated adjustment costs, and a relaxation cumulative value that is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of one pixel and disparity values ​​of pixels surrounding that one pixel by the cumulative value of that one pixel.a disparity value derivation element that derives a disparity value for each of the pixels based on the calculated relaxation accumulation value, and a disparity map generator that generates the disparity map based on the derived disparity value,

[0007] Another system and method for generating a disparity map are known from US 2015 / 296202 A1. Summary

[0008] One of the objectives of the present disclosure is to provide a method for generating a disparity map which can perform the comparison of stereo images even under bad weather conditions.

[0009] The present disclosure or invention provides a system for generating a disparity map according to claim 1 and a method for generating a disparity map according to claim 5. Advantageous embodiments are described in the dependent claims.

[0010] According to an exemplary embodiment of the present disclosure, a system for generating a disparity map comprises: an image acquisition element (e.g., an image recording element) that acquires a left image and a right image; a matching cost calculator that calculates matching costs (in other words, matching costs; e.g.,a disparity measure) for each of a plurality of pixels in the left image and the right image; a cumulative addition calculator that calculates a cumulative value for one of the pixels based on the calculated adjustment costs, and a relaxation cumulative value that is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of that one pixel and disparity values ​​of the pixels surrounding that one pixel by the cumulative value of that one pixel; a disparity value derivation element that derives a disparity value for each of the pixels based on the calculated relaxation cumulative value; and a disparity map generator that generates the disparity map based on the derived disparity value.

[0011] When the accumulation and addition calculator accumulates and adds the adjustment costs in a vertical direction, then, according to the invention, the accumulation and addition calculator calculates a current accumulation value by adding a minimum value of a accumulation value of a previous direction of the corresponding disparity, a value obtained by adding a first accumulation constant to a accumulation value of a direction around a disparity value, and a value obtained by adding a second accumulation constant to a minimum value of the accumulation values ​​of the previous direction, to current adjustment costs; and when the accumulation and addition calculator accumulates and adds the adjustment costs in a horizontal direction,Then, according to the invention, the accumulation and addition calculator calculates the current accumulation value by adding a minimum value of the accumulation value of the previous direction of the corresponding disparity and the value obtained by adding the second accumulation constant to the minimum value of the accumulation values ​​of the previous direction, to the current adjustment costs.

[0012] The accumulation and addition calculator can compute a repeating (e.g., recursive) relaxation accumulation value, which is an average of values ​​obtained by multiplying each of the ratio coefficients between the disparity value of one pixel and the disparity values ​​of pixels surrounding that one pixel by the relaxation accumulation value of that one pixel, and the disparity value derivation element can derive the disparity value for each of the pixels based on the repeating relaxation accumulation value.

[0013] The disparity value derivation element can derive a disparity value whose calculated cumulative value reaches a minimum as the disparity value of a single pixel.

[0014] The system may further include: a stixel-applying element that groups pixels that have the same disparity value in a vertical or horizontal direction to form a stixel, and changes the disparity values ​​of pixels located / configured between a plurality of stixels to disparity values ​​of the plurality of stixels when a pixel distance between the plurality of stixels that are formed / configured in the same direction, such that they are positioned on the same line and have the same disparity value, is a predetermined pixel value or less.

[0015] According to a further exemplary embodiment of the present disclosure, a method for generating a disparity map comprises: obtaining a left image and a right image; calculating adjustment costs for each of a plurality of pixels of the left image and the right image; calculating a cumulative value for one of the pixels based on the calculated adjustment costs; calculating a relaxation cumulative value, which is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of the one pixel and the disparity values ​​of pixels surrounding the one pixel by the cumulative value of the one pixel; deriving a disparity value for each of the pixels based on the calculated relaxation cumulative value; and generating the disparity map based on the derived disparity value.

[0016] When calculating the cumulative value, if the adjustment costs are accumulated and added in a vertical direction, a current cumulative value is calculated according to the invention by adding a minimum value of a cumulative value of a previous direction of the corresponding disparity, a value obtained by adding a first cumulative constant to a cumulative value of a direction around a disparity value, and a value obtained by adding a second cumulative constant to a minimum value of the cumulative values ​​of the previous direction, to the adjustment costs. If the adjustment costs are accumulated and added in a horizontal direction, the current cumulative value is calculated according to the invention by adding a minimum value of the cumulative value of the previous direction of the corresponding disparity and the value of the previous direction of the corresponding disparity.which is obtained by adding the second cumulative constant to the minimum value of the cumulative values ​​of the previous direction, to the current adjustment costs.

[0017] The method may further include: after calculating the relaxation accumulation value, calculating a repeating (e.g., recursive) relaxation accumulation value, which is an average of values ​​obtained by multiplying each of the ratio coefficients between the disparity value of the one pixel and the disparity values ​​of the pixels surrounding the one pixel by the relaxation accumulation value of the one pixel, wherein, when deriving the disparity value for each of the pixels, the disparity value for each of the pixels is derived based on the repeating relaxation accumulation value.

[0018] When deriving the disparity value for each of the pixels, a disparity value whose calculated cumulative value becomes a minimum can be derived as the disparity value of that one pixel.

[0019] The method may further include: grouping pixels that have the same disparity value in a vertical or horizontal direction to form a stixel, and changing disparity values ​​of the pixels located between a plurality of stixels to disparity values ​​of the plurality of stixels if a pixel distance between the plurality of stixels formed in the same direction, such that they are positioned on the same line and have the same disparity value, is a predetermined pixel value or less. Brief description of the illustrations

[0020] The above and other objectives, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings. Fig. Figure 1 is a block diagram of a system for generating a disparity map according to an exemplary embodiment of the present disclosure. Fig. Figure 2A is a diagram illustrating a process of accumulating and adding the adjustment costs from top to bottom in a vertical direction according to an exemplary embodiment of the present disclosure. Fig. Figure 2B is a diagram illustrating a process of accumulating and adding the adjustment costs from bottom to top in a vertical direction according to an exemplary embodiment of the present disclosure. Fig. Figure 3A is a diagram illustrating a process of accumulating and adding the adjustment costs from left to right in a horizontal direction according to an exemplary embodiment of the present disclosure. Fig. Figure 3B is a diagram illustrating a process of accumulating and adding the adjustment costs from right to left in a horizontal direction according to an exemplary embodiment of the present disclosure. Fig. Figure 4 is a diagram illustrating a process of relaxation accumulation and adding of adjustment costs according to an exemplary embodiment of the present disclosure. Fig. 5A is an adjustment cost map that represents adjustment costs for each of the pixels according to an exemplary embodiment of the present disclosure. Fig. 5B is a cumulative value map that represents a cumulative value for each of the pixels in the case that the cumulative and addition are performed once in one direction from left to right, according to an exemplary embodiment of the present disclosure. Fig. 5C is a cumulative value map that represents a cumulative value for each of the pixels in the case that the cumulative and additive operations are performed twice in a left-to-right direction, according to an exemplary embodiment of the present disclosure. Fig. 5D is a cumulative value map that represents a cumulative value for each of the pixels in the case that the cumulative and additive functions are performed three times in a left-to-right direction, according to an exemplary embodiment of the present disclosure. Fig. 6A is a cumulative value map from which an average of cumulative values ​​of the respective directions is derived, according to an exemplary embodiment of the present disclosure. Fig. 6B is a cumulative values ​​card in which relaxation accumulation and addition are shown once in the cumulative values ​​card of the Fig. 6A shall be carried out in accordance with an exemplary embodiment of the present disclosure. Fig. 6C is a cumulative value chart, in which relaxation accumulation and addition are shown again in the cumulative value chart of the Fig. 6B shall be carried out in accordance with an exemplary embodiment of the present disclosure. Fig. 7A is a cumulative value map that represents a relaxation cumulative value when a disparity value is 0, according to an exemplary embodiment of the present disclosure. Fig. 7B is a cumulative value chart that represents the relaxation cumulative value when the disparity value is 1, according to an exemplary embodiment of the present disclosure. Fig. 7C is a cumulative value chart that represents the relaxation cumulative value when the disparity value is 9, according to an exemplary embodiment of the present disclosure. Fig. Figure 8 is a block diagram representing a system for generating a disparity map according to another exemplary embodiment of the present disclosure. Fig. Figure 9 is a disparity map to which a stixel is applied, according to an exemplary embodiment of the present disclosure. Fig. Figure 10 is a flowchart that presents a method for generating a disparity map according to an exemplary embodiment of the present disclosure. Fig. Figure 11 is a flowchart that presents a method for generating a disparity map according to a further exemplary embodiment of the present disclosure. Fig. 12A is a left image obtained by means of an image acquisition element according to an exemplary embodiment of the present disclosure. Fig. 12B is a right-hand image obtained using the image acquisition element according to an exemplary embodiment of the present disclosure. Fig. 13 is a disparity map created by using the left image of the Fig. 12A and the right image of the Fig. 12B is produced according to the procedure for matching the stereo images according to the related technique. Fig. 14 is a disparity map created by using the left image of the Fig. 12 A and the right image of the Fig. 12 B is produced according to an exemplary embodiment of the present disclosure. Fig. Figure 15 is a block diagram illustrating a computer system performing a method for generating a disparity map according to an exemplary embodiment of the present disclosure. Detailed description

[0021] It is to be understood that the term "vehicle" or "vehicle-" or any other similar term, as used herein, includes motor vehicles in general, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, watercraft including a plurality of boats and ships, aircraft and the like, and hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles and other vehicles powered by alternative fuels (e.g., fuels derived from raw materials other than petroleum). Hereinafter, a hybrid vehicle means a vehicle having two or more sources of propulsion, for example, vehicles powered by both gasoline and electricity.

[0022] The terminology used herein serves only the purpose of describing certain embodiments and is not intended to limit the disclosure. As used herein, the singular forms "a," "an," "the," "a," and "a" are intended to include the plural forms unless the context clearly indicates otherwise. It is further understood that the terms "have" and / or "having" when used in this description specify the presence of specified features, numbers, steps, activities, elements, and / or components, but do not exclude the presence or addition of one or more other features, numbers, steps, activities, elements, components, and / or groups thereof. As used herein, the expression "and / or" includes any and all combinations of one or more of the related, listed elements.Throughout this description, unless explicitly stated otherwise, the word "include" and variations such as "indicates" or "indicating" will be understood to mean the inclusion of the specified elements, but not the exclusion of any other elements. Furthermore, the terms "unit," "-er," "-element," and "module" used in this description refer to units (e.g., devices) capable of performing at least one function and operation, and can be implemented by hardware components or by software components and combinations thereof.

[0023] Furthermore, the control logic of this disclosure can be contained as a durable, computer-readable medium on a computer-readable medium / data carrier, which contains executable program instructions that are executed by a processor, controller, or the like. Examples of computer-readable data carriers include, but are not limited to, ROM, RAM, compact disc (CD)-ROMs, magnetic tapes, floppy disks, flash memory, smart cards, and optical data storage devices. The computer-readable medium can also be distributed in networked computer systems, such that the computer-readable media are stored and executed in a distributed manner, e.g., through a telematics server or a controller area network (CAN).

[0024] Below, some exemplary embodiments of the present disclosure are described in detail with reference to the illustrative drawings. It should be noted that when assigning reference numerals to components in each of the accompanying drawings, the same components are given the same reference numerals, even if they are shown in different drawings. Furthermore, when describing the exemplary embodiments of the present disclosure, well-known designs or functions are not described in detail if they would unnecessarily complicate the understanding of the exemplary embodiments of the present disclosure.

[0025] Fig. Figure 1 is a block diagram of a system for generating a disparity map according to an exemplary embodiment of the present invention.

[0026] Referring to Fig. 1. A system 100 for generating a disparity map can comprise: an image acquisition element 110, an adjustment cost calculator 120, a cumulative addition calculator 130, a disparity value derivation element 140, and a disparity map generator 150. The image acquisition element 110 can be a camera or an image receiver. Furthermore, the adjustment cost calculator 120, the cumulative addition calculator 130, the disparity value derivation element 140, and the disparity map generator 150 can be one or more controllers, each containing a processor for performing the corresponding operations, and each can be contained in a durable, computer-readable medium.

[0027] However, the ones in Fig. The components shown in Figure 1 are not essential components. Therefore, System 100 can generate a disparity map with more or fewer components than shown in Figure 100. Fig. As shown in 1, it will be implemented.

[0028] Image acquisition element 110 can acquire a left image and a right image. Image acquisition element 110 can be a camera that records an image, and it can also be an image receiver that receives the image from the camera. If image acquisition element 110 is the camera, it can consist of two cameras separated horizontally. Furthermore, if image acquisition element 110 is the image receiver, the camera that records the received image can consist of two cameras arranged horizontally.

[0029] In principle, even if the two cameras, which are separated from each other in a horizontal direction, capture the image in the same direction, the images obtained from the respective cameras have a difference due to a spatial distance between the two cameras, and the system 100 for generating a disparity map according to the present disclosure generates the disparity map based on the aforementioned difference.

[0030] The matching cost calculator 120 can calculate the matching costs for each pixel of the left and right images. According to the present disclosure, the matching cost calculator 120 can use a ground-based stereo image matching algorithm. The ground-based stereo image matching algorithm is an algorithm that calculates the matching costs by comparing the ground images (e.g., subsurface images, such as floor images) of the left and right images, and the matching costs by comparing the images above the ground of the left and right images.According to the algorithm for matching stereo images based on the ground, since a ground disparity marker and an object disparity marker are calculated in such a way that they are separate from each other, the disparity map can be generated in such a way that it is more robust with respect to the ground than a semiglobal method according to the related technique.

[0031] The Accumulation and Addition Calculator 130 can calculate a cumulative value for each of the pixels based on the calculated adjustment costs, and can perform a relaxation accumulation and an addition based on cumulative values ​​around the pixel to calculate a relaxation accumulation value for each of the pixels.

[0032] First, a procedure for calculating a cumulative value in a vertical direction and a horizontal direction using the cumulative and addition calculator 130 is described with reference to Fig. 2A to 3B described.

[0033] Fig. Figure 2A is a diagram illustrating a process of accumulating and adding the adjustment costs from top to bottom in a vertical direction according to an exemplary embodiment of the present disclosure. Fig. Figure 2B is a diagram illustrating a process of accumulating and adding the adjustment costs from bottom to top in a vertical direction according to an exemplary embodiment of the present disclosure.

[0034] Fig. Figure 3A is a diagram illustrating a process of accumulating and adding the adjustment costs from left to right in a horizontal direction according to an exemplary embodiment of the present disclosure. Fig. Figure 3B is a diagram illustrating a process of accumulating and adding the adjustment costs from right to left in a horizontal direction according to an exemplary embodiment of the present disclosure.

[0035] In the case where adjustment costs are accumulated and added in the vertical direction, the accumulation and addition calculator can compute a current accumulation value by adding a minimum value of a accumulation value of a previous direction of the corresponding disparity, a value obtained by adding a first accumulation constant to a accumulation value of a direction around a disparity value, and a value obtained by adding a second accumulation constant to a minimum value of the accumulation value of the previous direction, to the current adjustment cost.

[0036] This can be represented in the following equation 1: LR(p,d)=C(p,d)+min(LR(p−r,d),LR(p−r,d−1)+p1,LR(p−r,d+1)+p1,min(LR(p−r,i)+p2))

[0037] The expression L R (p, d) is the cumulative value, p1 is the first cumulative constant, p2 is the second cumulative constant, C(p, d) is the adjustment cost, p is a pixel coordinate, d is the disparity value, and r is a direction of cumulative addition.

[0038] Fig. 2A represents a process of accumulating and adding adjustment costs in one direction, from top to bottom, and Fig. 2B represents a process of accumulating and adding adjustment costs in one direction, from bottom to top.

[0039] As in Fig. 2A and Fig. As shown in Figure 2B, the accumulation and addition calculator 130 can calculate the current accumulation value by adding the minimum value of the accumulation value of the previous direction of the corresponding disparity, the value obtained by adding the first accumulation constant to the accumulation value of the direction around the disparity value, and the value obtained by adding the second accumulation constant to the minimum value of the accumulation value of the previous direction, to the current adjustment cost.

[0040] The cumulative constants are arbitrary constants that are added to the cumulative value of the previous direction or to the cumulative value of the direction around the disparity value, and the cumulative constants are added, thus making it possible to strengthen the vertical relationship.

[0041] Furthermore, in the case that the adjustment costs are accumulated and added in the horizontal direction, the accumulation and addition calculator 130 can calculate the current accumulation value by adding the minimum value of the accumulation value of the previous direction of the corresponding disparity and the value obtained by adding the second accumulation constant to the minimum value of accumulation values ​​of the previous direction to the current adjustment costs.

[0042] This can be represented in the following equation 2: LR(p,d)=C(p,d)+min(LR(p−r,d),min(LR(p−r,i)+p2)

[0043] The expression L R (p, d) is the cumulative value, p2 is the second cumulative constant, C(p, d) is the adjustment cost, p is a pixel coordinate, d is the disparity value, and r is a direction of cumulative addition.

[0044] Fig. 3A represents a process of accumulating and adding the adjustment costs in one direction from left to right, and Fig. 3B represents a process of accumulating and adding the adjustment costs in one direction, from right to left.

[0045] As in Fig. 3A and Fig. As shown in Figure 3B, the accumulation and addition calculator 130 can calculate the current accumulation value by adding the minimum value of the accumulation value of the previous direction of the corresponding disparity and the value obtained by adding the second accumulation constant to the minimum value of the accumulation value of the previous direction to the current adjustment cost.

[0046] Unlike accumulation and addition in the vertical direction, when accumulation and addition are performed in the horizontal direction, only the second accumulation constant is used, and the accumulation constant is added, thus making it possible to strengthen a horizontal relationship.

[0047] The Accumulation and Addition Calculator 130 can perform a relaxation, accumulation, and addition calculation based on the accumulation values, which, as described above, are accumulated and added in the vertical and horizontal directions. The relaxation, accumulation, and addition calculation is based on the accumulation values ​​around a current pixel, and a relationship between the current pixel and the surrounding pixels can be strengthened by performing the relaxation, accumulation, and addition calculation. A procedure for performing a relaxation, accumulation, and addition calculation is described below with reference to... Fig. 4 and equation 3 are described.

[0048] Fig. Figure 4 is a diagram illustrating a process of relaxation accumulation and addition of adjustment costs according to an exemplary embodiment of the present disclosure.

[0049] The Cumulative and Additive Calculator 130 can multiply ratio coefficients between the disparity value of the current pixel and the disparity values ​​of the pixels surrounding the current pixel by the cumulative value of the current pixel, and calculate an average of the multiplied values ​​(in other words, the values ​​obtained by the multiplication) as a relaxation cumulative value. Furthermore, the Cumulative and Additive Calculator 130 can repeat the relaxation cumulative and additive calculations based on the calculated relaxation cumulative value.This means that the accumulation and addition calculator 130 can multiply the ratio coefficients between the disparity value of the current pixel and the disparity values ​​of the pixels surrounding the current pixel by the relaxation accumulation value of the current pixel and calculate an average of the multiplied values ​​(in other words, the values ​​obtained by the multiplication) as a relaxation-repeating (e.g., recursive) relaxation accumulation value. Here, the ratio coefficient is (an estimated disparity of a current pixel - an estimated disparity of the surrounding pixels) / (a total permissible range of disparity values).

[0050] This can be represented in the following equation 3. Ln(p,d)=1n(r)∑rLrn(p,d)×q(p,p−r)

[0051] The expression L r n(p, d) is a cumulative value repeated n times, n(r) is the number of values ​​surrounding the current pixel, q(p, pr) is the relation coefficient between the disparity value of the current pixel and the disparity values ​​of the surrounding pixels, and r is a direction of the cumulative addition, d is the disparity value, and pr is a previous value of the current pixel in the r-direction.

[0052] An initial value of the cumulative value (L r n (p,d)) are the adjustment costs (L r 0 (p,d))=C(p, d)), and a value obtained by multiplying the ratio coefficients between the disparity value of the current pixel and the disparity values ​​of the surrounding pixels by the adjustment costs and dividing the result by multiplying the obtained value by the number of surrounding cumulative values ​​is the relaxation cumulative value.

[0053] The accumulation and addition calculator 130 can perform the repetitive relaxation-accumulation and addition calculation by substituting the relaxation-accumulation value, which is known as the accumulation value (L). r n (p,d)) is calculated, perform.

[0054] Disparity Value Derivation Element 140 can derive a disparity value for each pixel based on the calculated cumulative value. Disparity Value Derivation Element 140 can derive a disparity value whose calculated cumulative value is at its minimum as the disparity value of the current pixel. In Equation 3, the quantity d, representing the disparity value, is included, and Disparity Value Derivation Element 140 derives the quantity d that minimizes the cumulative value as the disparity value of the current pixel.

[0055] The disparity map generator 150 can generate the disparity map based on the derived disparity value. This means that the disparity value derivation element 140 can derive disparity values ​​for all pixels of an image, and the image displaying the disparity values ​​for each pixel becomes the disparity map.

[0056] Below is an example of a process for calculating the cumulative value based on the adjustment costs calculated using the Adjustment Cost Calculator 120, and deriving the disparity value with reference to Fig. 5A to 7C described.

[0057] Fig. 5A is an adjustment cost map that represents adjustment costs for each of the pixels according to an exemplary embodiment of the present disclosure. Fig. 5B is a cumulative value map that represents a cumulative value for each of the pixels in the case that the cumulative and additive functions are performed once in a left-to-right direction according to an exemplary embodiment of the present disclosure. Fig. 5C is a cumulative value map that represents a cumulative value for each of the pixels in the case that the cumulative and additive functions are performed twice in a left-to-right direction according to an exemplary embodiment of the present disclosure. Fig. 5D is a cumulative value map that represents a cumulative value for each of the pixels in the case that the cumulative and additive functions are performed three times in a left-to-right direction according to an exemplary embodiment of the present disclosure.

[0058] Fig. 6A is a cumulative value map from which an average of cumulative values ​​of the respective directions is derived, according to an exemplary embodiment of the present disclosure. Fig. 6B is a cumulative values ​​card in which relaxation accumulation and addition are shown once in the cumulative values ​​card of Fig. 6A is carried out according to an exemplary embodiment of the present disclosure. Fig. 6C is a cumulative values ​​chart in which the relaxation-cumulative and addition calculations are repeated in the cumulative values ​​chart of Fig. 6B is carried out according to an exemplary embodiment of the present disclosure.

[0059] Fig. 7A is a cumulative value map that represents a relaxation cumulative value when a disparity value is 0, according to an exemplary embodiment of the present disclosure. Fig. 7B is a cumulative value chart that represents the relaxation cumulative value when the disparity value is 1, according to an exemplary embodiment of the present disclosure. Fig. 7C is a cumulative value chart that represents the relaxation cumulative value when the disparity value is 9, according to an exemplary embodiment of the present disclosure.

[0060] Referring to Fig. 5A represents the adjustment cost in all pixels 1 if the disparity is 1. The accumulation and addition calculator 130 performs the accumulation and addition calculation for the respective directions, and Fig. 5B to 5D represent a process of performing a cumulative and additive calculation from left to right in a horizontal direction.

[0061] As described above, since the initial value of the cumulative value is equal to the adjustment costs, the cumulative value of the pixels in the leftmost column is in Fig. 5B (equal to) 1.

[0062] The Accumulation and Addition Calculator 130 can calculate a cumulative value for a second column using equation 2. The Accumulation and Addition Calculator 130 derives a cumulative value of 2, as in Fig. 5C is represented by adding a value obtained by adding the second cumulative constant to the minimum value of the cumulative values ​​of the previous directions of all disparity indicators and (the value) 1, which is the minimum of the cumulative values ​​of the previous directions with a current disparity indicator, to the value 1, which represents the current adjustment costs.

[0063] The accumulation and addition calculator 130 can recalculate a cumulative value of a third column using equation 2. The accumulation and addition calculator 130 derives a cumulative value of 3, as in Fig. 5D represented, by adding a value obtained by adding the second cumulative constant to the minimum value of the cumulative values ​​of the previous directions of all disparity indicators and the value 2, which is the minimum of the cumulative values ​​of the previous directions with the current disparity indicator, to the value 1, which represents the current adjustment costs.

[0064] The accumulation and addition calculator 130 can calculate the accumulation values ​​of all directions and the relaxation accumulation value according to the procedure described above. Fig. 6A represents the cumulative values ​​that are determined using the cumulative and addition calculator 130 according to the procedure described above.

[0065] This means that the accumulation and addition calculator 130 displays the current accumulation value of the middle pixel, which is the current pixel in Fig. 6A is multiplied according to equation 3 and the relaxation accumulation value as in Fig. 6B is derived by dividing the value obtained by multiplication by the number of surrounding cumulative values.

[0066] The accumulation and addition calculator 130 can re-perform the relaxation-accumulation and addition calculation for the relaxation-accumulation value obtained from equation 3. A result obtained by re-performing the relaxation-accumulation and addition calculation is shown in Fig. 6C shown, and the number of repetitions at which the cumulative and additive calculator 130 can perform the cumulative and additive calculation is not limited to the above representation.

[0067] The disparity value derivation element 140 can derive a disparity value whose relaxation accumulation value, calculated using the accumulation-addition calculator 130, is at a minimum, as the disparity value of the current pixel. (In other words, the disparity value of the current pixel can be the one at which the relaxation accumulation value is minimal.)

[0068] Fig. 7A represents the relaxation accumulation value when the disparity value is 0, Fig. 7B represents the relaxation accumulation value when the disparity value is 1, and Fig. 7C represents the relaxation accumulation value when the disparity value is 9. Referring to Fig. 7A to 7C, if the disparity value is 9, the relaxation accumulation value is 3, which is the minimum value, and the disparity value derivation element 140 can derive that the disparity value is 9.

[0069] Meanwhile, a stixel is used according to another exemplary embodiment of the present disclosure to increase the coherence between pixels, thereby making it possible to eliminate gaps that may be present in the disparity maps and to generate the disparity map which has a strengthened coherence (e.g. smoother transitions between the individual values).

[0070] Fig. Figure 8 is a block diagram of a system for generating a disparity map according to a further exemplary embodiment of the present disclosure.

[0071] Referring to Fig. 8 A system 200 for generating a disparity map according to a further exemplary embodiment of the present disclosure may comprise an image acquisition element 210, an adjustment cost calculator 220, a cumulation and addition calculator 230, a disparity value derivation element 240, a disparity map generator 250 and a Stixel-applying element 260.

[0072] However, the ones in Fig. The 8 components shown are not essential components. Therefore, the device 200 can be used to generate a disparity map with more or fewer components than shown in Fig. As shown in section 8, it will be implemented.

[0073] Since the image acquisition element 210, the adjustment cost calculator 220, the accumulation and addition calculator 230, the disparity value derivation element 240, and the disparity map generator 250 are the same as those referred to in Fig. In the case of the section described in point 1, a description of it will be omitted.

[0074] The Stixel-applying element 260 can group pixels that have the same disparity value in the vertical or horizontal direction to form / configure a stixel. If the pixel distance between a plurality of stixels formed / configured in the same direction, positioned on the same line, and with the same disparity value is a predetermined pixel value or less, the Stixel-applying element 260 can change the disparity values ​​of pixels located between the plurality of stixels to the disparity values ​​of the plurality of stixels.

[0075] Element 260, which applies stixels, can group pixels that have the same disparity value in either the vertical or horizontal direction to form / configure a stixel. If the disparity value is derived normally, a base image will have the same disparity value in the horizontal direction and will contain a stixel.

[0076] However, since it is difficult to derive the disparity value normally under bad weather conditions, some parts of the basic image have the same disparity value in the horizontal direction, while other parts have different disparity values. Therefore, the basic image cannot contain (only) one stixel.

[0077] In the event that the pixel distance between the plurality of stixels formed / configured in the same direction, such that they are positioned on the same line and have the same disparity value, is a predetermined pixel value or less, the stixel-applying element 260 can change the disparity values ​​of pixels located between the plurality of stixels to the disparity values ​​of the plurality of stixels.

[0078] In the event that the majority of stixels are configured in the same direction, so that they are positioned on the same line, and the distance between the majority of stixels is small, it can be interpreted that pixels with different disparity values ​​located between the majority of stixels have erroneous disparity values ​​due to the weather. By taking the above aspect into account, the stixel-applying element 260 can perform a correction in which the disparity value of the pixels with the erroneous disparity value is changed to the disparity value of the majority of stixels.

[0079] Fig. Figure 9 is a disparity map to which a stixel is applied, according to an exemplary embodiment of the present disclosure.

[0080] If the disparity value is corrected by applying the stixel, as described above, the pixels between the plurality of stixels will have the same disparity value as the plurality of stixels. Therefore, the plurality of stixels can contain a stixel, as in Fig. 9 shown.

[0081] The following is a method for generating a disparity map based on the arrangements described above, with reference to Fig. 10 described.

[0082] Fig. Figure 10 is a flowchart that presents a method for generating a disparity map according to an exemplary embodiment of the present disclosure.

[0083] Referring to Fig. 10 may include a method for generating a disparity map according to an exemplary embodiment of the present disclosure: an operation (S110) of obtaining a left image and a right image, an operation (S120) of calculating adjustment costs of the left image and the right image, an operation (S130) of calculating a cumulative value for each of the pixels based on the adjustment costs, an operation (S140) of calculating a relaxation cumulative value based on the cumulative value for each of the pixels, an operation (S150) of deriving a disparity value for each of the pixels based on the relaxation cumulative value, and an operation (S160) of generating the disparity map based on the disparity value.

[0084] The operations (S110 to S160) are described in detail below.

[0085] In operation (S110), an image acquisition element 110 can acquire a left image and a right image. The left image and the right image are images recorded by two cameras, respectively, which are separated horizontally. The image acquisition element 110 can directly capture the image as a camera and can (alternatively) receive the image from the camera as an image receiver.

[0086] In operation (S120), the matching cost calculator 120 can calculate the matching costs for each pixel of the left and right images according to an algorithm for matching the stereo images based on a ground. As described above, the matching cost calculator 120 can calculate the matching costs by comparing the ground images of the left and right images, and it can also calculate the matching costs by comparing the images over the ground in the left and right images.

[0087] In operation (S130), the Cumulative Addition Calculator 130 can calculate the cumulative value for each pixel based on the adjustment costs. The Cumulative Addition Calculator 130 can calculate the cumulative value in both the vertical and horizontal directions, as described above.

[0088] In operation (S140), the Cumulative Addition Calculator 130 can perform a relaxation cumulative addition calculation based on the cumulative value to compute a relaxation cumulative value. The Cumulative Addition Calculator 130 can calculate the relaxation cumulative value, which is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of a current pixel and disparity values ​​of the surrounding pixels by the cumulative value of the current pixel. Since a detailed procedure for calculating a cumulative value has been described above, a description of this procedure is omitted. Furthermore, the Cumulative Addition Calculator 130 can calculate the relaxation cumulative value recursively based on the previously calculated relaxation cumulative value.

[0089] In operation (S150), the disparity value derivation element 140 can derive the disparity value for each pixel based on the relaxation accumulation value. Disparity value derivation element 140 can derive a disparity value for which the calculated relaxation accumulation value is at its minimum, as the disparity value of the current pixel.

[0090] In operation (S160), the disparity map generator 150 can generate the disparity map based on the derived disparity value. The image, in which the disparity value for each pixel is represented, becomes the disparity map.

[0091] Below is a method for generating a disparity map according to a further exemplary embodiment of the present disclosure with reference to Fig. 11 described.

[0092] Fig. Figure 11 is a flowchart that presents a method for generating a disparity map according to a further exemplary embodiment of the present disclosure.

[0093] Referring to Fig. 11 The method for generating a disparity map according to a further exemplary embodiment of the present disclosure may comprise: an operation (S210) of obtaining a left image and a right image, an operation (S220) of calculating adjustment costs of the left image and the right image, an operation (S230) of calculating a cumulative value for each of the pixels based on the adjustment costs, an operation (S240) of calculating a relaxation cumulative value based on the cumulative value for each of the pixels, an operation (S250) of deriving a disparity value for each of the pixels based on the cumulative value, an operation (S260) of correcting the disparity value by applying a stixel, and an operation (S270) of generating the disparity map based on the disparity value.

[0094] Since the operations (S210 to S250 and S270) are the same as the operations described above (S110 to S160), a description of them is omitted.

[0095] In operation (S260), a stixel-applying element can group 260 pixels that have the same disparity value in either the vertical or horizontal direction to form / configure a stixel. If the pixel distance between multiple stixels formed / configured in the same direction, positioned on the same line, and with the same disparity value is a predetermined pixel value or less, a correction can be performed in which the disparity values ​​of the pixels located between the multiple stixels are changed to the disparity values ​​of the multiple stixels.

[0096] By adding the operation (S260), gaps that may occur in the disparity map which uses an image taken in bad weather are removed, thus making it possible to obtain the disparity map with enhanced correlation.

[0097] Fig. 12A is a left image obtained by an image acquisition element according to an exemplary embodiment of the present disclosure. Fig. 12B is a right-hand image obtained by the image-gathering element according to an exemplary embodiment of the present disclosure.

[0098] Fig. 13 is a disparity map that, when using the left image of the Fig. 12A and the right image of the Fig. 12B was created using the procedure for matching stereo images according to the related technique. Fig. 14 is a disparity map, which, when using the left image of the Fig. 12A and the right image of the Fig. 12B is produced according to an exemplary embodiment of the present disclosure.

[0099] Fig. 12A and Fig. Figure 12B represents the left and right images that can be obtained by the image acquisition element of the device 100 or 200 for generating a disparity map according to the present disclosure. Referring to Fig. 12A and Fig. 12B confirms that the picture was taken in bad, rainy weather.

[0100] According to a method for matching stereo images according to the related technique (an image matching based on a local matching, an image matching based on a semi-global matching, etc.), it can be confirmed that erroneous disparity values ​​are derived in the disparity map, as in Fig. Figure 13 shows how the gaps arise.

[0101] However, in the case that the disparity map is generated by the arrangement and method according to the present disclosure as described above, it can be confirmed that the disparity is consistently shown horizontally to a road surface, and the disparity is shown vertically to the object, as in Fig. 14 shown, which makes it possible to generate an excellent disparity map even in bad weather.

[0102] Fig. Figure 15 is a block diagram representing a computer system that performs the method for generating a disparity map according to an exemplary embodiment of the present disclosure.

[0103] Referring to Fig.15 A computer system 1000 can have at least one processor 1100, one memory 1300, one user interface input device 1400, one user interface output device 1500, one storage 1600 and one network interface 1700, which are connected by a bus 1200.

[0104] The 1100 processor can be a central processing unit (CPU) or a semiconductor device that executes processes for program instructions stored in memory 1300 and / or memory 1600. Memory 1300 and memory 1600 can have different types of volatile or non-volatile storage media. For example, memory 1300 can have read-only memory (ROM) and random-access memory (RAM).

[0105] Accordingly, operations in the method or algorithm disclosed in connection with exemplary embodiments in this description can be implemented directly in hardware, a software module, or a combination thereof, executed by the processor 1100. The software module can be located on a storage medium (e.g., the memory 1300 and / or the memory unit 1600), such as random-access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable storage medium, or a compact disc read-only memory (CD-ROM).An illustrative storage medium can be coupled to the Processor 1100, and the Processor 1100 can read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the Processor 1100. The processor and the storage medium can also be located within an application-specific integrated circuit (ASIC). The ASIC can be located within a user terminal (e.g., a user end device). Alternatively, the processor and the storage medium can also be located within a user terminal as a single, separate component.

[0106] In the system and method for generating a disparity map as described above, the arrangement and method of the exemplary embodiments described above cannot be applied restrictively. That is to say, all or some of the individual exemplary embodiments can be configured to be selectively combined with one another, so that they can be modified in various ways.

[0107] As described above, according to the exemplary embodiments of the present disclosure, it is possible to provide the method for generating a disparity map which is capable of performing the alignment of stereo images excellently even under bad weather conditions. Reference symbol for each of the elements in the drawings 110 Image Acquisition Element 120 Adjustment Cost Calculator 130 Cumulative and Additive Calculators 140 Disparity value derivative element 150 disparity map producers 210 Image Acquisition Element 220 Adjustment Cost Calculator 230 Cumulative and Additive Calculators 240 Disparity value derivative element 260 Stixel-using element 250 disparity map producers 1100 processor 1300 storage 1400 User interface input device 1500 User Interface Output Device 1600 storage 1700 network interface

Claims

[1] System (100, 200) for generating a disparity map, the system comprising: an image acquisition element (110, 210) that acquires a left image and a right image; an adjustment cost calculator (120, 220) that calculates adjustment costs for each of a plurality of pixels of the left image and the right image, a cumulative and additive calculator (130, 230) that calculates a cumulative value of one of the pixels based on the calculated adjustment costs and calculates a relaxation cumulative value that is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of one pixel and disparity values ​​of pixels surrounding that one pixel by the cumulative value of that one pixel, a disparity value derivation element (140, 240) that derives a disparity value for each of the pixels based on the calculated relaxation accumulation value, and a disparity map generator (150, 250) that generates the disparity map based on the derived disparity value, wherein, when the cumulative and additive calculator (130, 230) cumulatively and additively calculates the adjustment costs in a vertical direction, the cumulative and additive calculator (130, 230) calculates a current cumulative value by adding a minimum value of a cumulative value of a previous direction around the corresponding disparity, a value obtained by adding a first cumulative constant to a cumulative value of a direction around a disparity value, and a value obtained by adding a second cumulative constant to a minimum value of the cumulative values ​​of the previous direction, to current adjustment costs, and When the cumulative and additive calculator (130, 230) cumulatively and additively calculates the adjustment costs in a horizontal direction, the cumulative and additive calculator (130, 230) calculates the current cumulative value by adding a minimum value of the cumulative value of the previous direction of the corresponding disparity and the value obtained by adding the second cumulative constant to the minimum value of the cumulative values ​​of the previous direction, to the current adjustment costs. [2] System (100, 200) according to claim 1, wherein the accumulation and addition calculator (130, 230) calculates a repeating accumulation value which is an average of the values ​​obtained by multiplying each of the ratio coefficients between the disparity value of one pixel and the disparity values ​​of the pixels surrounding one pixel by the relaxation accumulation value of one pixel, and the disparity value derivation element (140, 240) derives the disparity value for each of the pixels based on the repeating relaxation accumulation value. [3] System (100, 200) according to one of claims 1 or 2, wherein the disparity value derivation element (140, 240) derives a disparity value whose calculated cumulative value assumes a minimum as the disparity value of one pixel. [4] System (200) according to any one of claims 1 to 3, further comprising: a stixel-applying element (260) that groups pixels that have the same disparity value in a vertical direction or a horizontal direction to configure a stixel, and changes the disparity values ​​of pixels located between a plurality of stixels to the disparity values ​​of the plurality of stixels when a pixel distance between the plurality of stixels configured in the same direction, such that they are positioned on the same line and have the same disparity value, is a predetermined pixel value or less. [5] Method for generating a disparity map, comprising the method: Obtained (S110, S210) by an image acquisition element (110, 210), a left image and a right image; Calculate (S120, S220), by an adjustment cost calculator (120, 220), of adjustment costs for each of a plurality of pixels of the left image and the right image; Calculate (S130, S230), by a cumulative and additive calculator (130, 230), a cumulative value of one of the pixels based on the calculated adjustment costs, Calculate (S140, S240), by the cumulation and addition calculator (130, 230), a relaxation cumulative value which is an average of values ​​obtained by multiplying each of the ratio coefficients between a disparity value of the one pixel and disparity values ​​of pixels surrounding the one pixel by the cumulative value of the one pixel, Deriving (S150, S250), through a disparity value derivation element (140, 240), a disparity value for each of the pixels based on the calculated relaxation accumulation value, and Generate (S160, S270) by a disparity map generator (150, 250) that generates a disparity map based on the derived disparity value. where, when calculating the cumulative value, if the adjustment costs are cumulative and added in a vertical direction, a current cumulative value is calculated by adding a minimum value of a cumulative value of a previous direction of the corresponding disparity, a value obtained by adding a first cumulative constant to a cumulative value of a direction around a disparity value, and a value obtained by adding a second cumulative constant to a minimum value of the cumulative values ​​of the previous direction, to current adjustment costs, and When adjustment costs are accumulated and added in a horizontal direction, the current accumulation value is calculated by adding a minimum value of the accumulation value of the previous direction of the corresponding disparity and the value obtained by adding the second accumulation constant to the minimum value of the accumulation values ​​of the previous direction to the current adjustment costs. [6] Method according to claim 5, further comprising: after calculating the relaxation accumulation value, calculating (S140, S240) a repeating relaxation accumulation value, which is an average of values ​​obtained by multiplying each of the ratio coefficients between the disparity value of the one pixel and the disparity values ​​of pixels surrounding the one pixel by the relaxation accumulation value of the one pixel, wherein when deriving (S150, S250) the disparity value for each of the pixels, the disparity value for each of the pixels is derived based on the repeating relaxation accumulation value. [7] Method according to one of claims 5 or 6, wherein when deriving (S150, S250) the disparity value for each of the pixels, a disparity value whose calculated cumulative value assumes a minimum is derived as the disparity value of one pixel. [8] Method according to any one of claims 5 to 7, further comprising: Grouping pixels that have the same disparity value in a vertical or horizontal direction to configure a stixel, and changing (S260) the disparity values ​​of pixels that are between a plurality of stixels to disparity values ​​of the plurality of stixels when a pixel distance between the plurality of stixels configured in the same direction so that they are positioned on the same line and have the same disparity value is a predetermined pixel value or less.

Images