Image stereo matching method and device

Abstract

The present application discloses a stereo matching method and device for an image. The specific implementation scheme is as follows: the method includes: processing the acquired image to obtain a disparity map and an invalid point area on the disparity map, the invalid point area including a first occlusion area; selecting an invalid point in the first occlusion area, and querying the first valid point and the second valid point with the smallest distance from the invalid point on both sides of the straight line passing through the invalid point, and the corresponding first disparity value and second disparity value; when the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset value, all the pixel points between the first valid point and the second valid point belong to the second occlusion area; performing disparity filling on the non-occlusion area in the second occlusion area and the invalid point area to obtain a dense disparity map. The occlusion area and the mismatch area are distinguished more accurately, so that the invalid points are filled more effectively, thereby improving the stereo matching effect of the image.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to the field of stereo matching. Background Art

[0002] The depth camera based on binocular stereo vision is similar to human eyes. It does not actively project light to the outside, but relies entirely on two pictures (color RGB or grayscale) to calculate the depth. The depth measurement process of the binocular stereo vision depth camera is as follows: 1. First, the binocular camera needs to be calibrated to obtain the internal and external parameters and homography matrix of the two cameras. 2. According to the calibration results, the original image is corrected. The two corrected images are located in the same plane and parallel to each other. 3. The two corrected images are matched pixel by pixel. 4. The depth of each pixel is calculated based on the matching results to obtain a depth map.

[0003] Stereo matching is a key part in stereo vision research. The goal is to find matching corresponding points from different viewpoint images. Specifically, the corresponding pixels are matched in two or more viewpoint images, the disparity is calculated, and an energy cost function is established and minimized to estimate the disparity of the pixels and obtain the depth. The stereo matching algorithm can be divided into four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. The steps of the local matching algorithm generally include matching cost calculation, matching cost aggregation, and disparity calculation. The global matching algorithm includes matching cost calculation, disparity calculation, and disparity optimization. The semi-global algorithm SGBM has all four steps. The complexity of the semi-global algorithm SGBM (Semi-Global Matching and) is relatively low, which is convenient for hardware implementation. However, the matching effect of the disparity map or depth map is poor. Summary of the invention

[0004] The present application provides a method and device for stereo matching of images to solve the problems existing in the related art. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for stereo matching of images, comprising:

[0006] Processing the acquired image to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area;

[0007] Select an invalid point in the first occlusion area, and query the first valid point and the second valid point with the smallest distance from the invalid point on both sides of the invalid point on a straight line passing through the invalid point, and the corresponding first disparity value and second disparity value respectively;

[0008] When the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset value, all pixel points between the first effective point and the second effective point belong to the second occlusion area;

[0009] The second occlusion area and the non-occlusion area in the invalid point area are filled with disparity to obtain a dense disparity map.

[0010] In one implementation, on a straight line passing through an invalid point, for both sides of the invalid point, respectively searching for a first valid point and a second valid point with the smallest distance from the invalid point, including:

[0011] When the straight line passing through the invalid point is a horizontal line, the first valid point and the second valid point with the smallest distance from the invalid point are queried for the left and right sides of the invalid point, respectively, as well as the corresponding first disparity value and second disparity value.

[0012] In one implementation, on a straight line passing through an invalid point, for both sides of the invalid point, respectively searching for a first valid point and a second valid point with the smallest distance from the invalid point, including:

[0013] When the straight line passing through the invalid point is a vertical line, the first valid point and the second valid point with the smallest distance from the invalid point, and the corresponding first disparity value and second disparity value are queried respectively for the upper side and the lower side of the invalid point.

[0014] In one implementation, performing parallax filling on the second occlusion area includes:

[0015] For multiple straight lines passing through the invalid point in the second occlusion area, valid points with the smallest distances and corresponding disparity values ​​are queried on the multiple straight lines respectively, and the angles between the multiple straight lines are the same;

[0016] A minimum disparity value is selected from the multiple disparity values ​​queried, and the minimum disparity value is used to perform disparity filling on invalid points in the second occlusion area.

[0017] In one implementation, processing the acquired image to obtain a disparity map includes:

[0018] Preprocess the acquired image, calculate the matching cost, fuse the matching cost, calculate the disparity, and obtain the disparity map;

[0019] The preprocessing step includes: performing Sobel processing on the acquired image to obtain a gradient value;

[0020] The steps of matching cost calculation include: performing BT processing on the gradient value to obtain the first generation value, performing BT processing on the acquired image to obtain the second generation value; performing Census transformation processing on the acquired image to obtain the third generation value;

[0021] The steps of matching cost fusion include: weighted addition of the first generation value, the second generation value, and the third generation value to obtain the target generation value.

[0022] In one implementation, weighted addition of the first generation value, the second generation value, and the third generation value to obtain a target cost value includes:

[0023] C=αC SobelX +β*C BT +γ*C census

[0024] Among them, C is the target cost value, C SobelX is the first generation value, C BT is the second generation value, C census It is the third generation value, α, β, and γ are all weight coefficients, and their value range is 0 to 1.

[0025] In one implementation, the non-occluded area is a mismatched area, and disparity filling is performed on the non-occluded area in the invalid point area, including:

[0026] Taking the invalid point in the mismatched area as the starting point, when valid points with the smallest distance are found in at least two directions, linear interpolation is performed using the disparity values ​​corresponding to the multiple valid points found;

[0027] The interpolation results are used to perform disparity filling on invalid points in the mismatched area.

[0028] In one embodiment, it further includes:

[0029] Taking the invalid point in the mismatching area as the starting point, if no valid point with the shortest distance is found along at least two directions, median calculation is performed using the disparity values ​​of the valid points around the invalid point in the mismatching area;

[0030] The median calculation result is used to perform disparity filling on invalid points in the mismatched area.

[0031] In a second aspect, the present application also provides a stereo matching device for images, including:

[0032] An image processing module, used to process the acquired image to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area;

[0033] A valid point query module is used to select an invalid point in the first occlusion area, and query a first valid point and a second valid point with the smallest distance from the invalid point on both sides of the straight line passing through the invalid point, and the corresponding first disparity value and second disparity value respectively;

[0034] an occlusion area determination module, configured to determine that, when an absolute value of a difference between a first disparity value and a second disparity value is greater than a preset value, all pixel points between the first effective point and the second effective point belong to a second occlusion area;

[0035] The disparity filling module is used to perform disparity filling on the non-occluded area in the second occluded area and the invalid point area to obtain a dense disparity map.

[0036] In one embodiment, the effective point query module includes:

[0037] The first query submodule is used to query the first and second valid points with the smallest distance from the invalid point, and the corresponding first and second disparity values, respectively, on the left and right sides of the invalid point when the straight line passing through the invalid point is a horizontal line.

[0038] In one embodiment, the effective point query module includes:

[0039] The second query submodule is used to query the first and second valid points with the smallest distance from the invalid point, and the corresponding first and second disparity values, respectively, on the upper and lower sides of the invalid point when the straight line passing through the invalid point is a vertical line.

[0040] In one embodiment, the parallax filling module includes:

[0041] A third query submodule is used to query the valid points with the shortest distances and the corresponding disparity values ​​on the multiple straight lines passing through the invalid point in the second occlusion area, and the angles between the multiple straight lines are the same;

[0042] The first filling submodule is used to select a minimum disparity value from the multiple disparity values ​​queried, and use the minimum disparity value to perform disparity filling on invalid points in the second occlusion area.

[0043] In one embodiment, the image processing module includes:

[0044] The disparity map generation submodule is used to preprocess the acquired image, perform matching cost calculation, matching cost fusion, and disparity calculation to obtain a disparity map;

[0045] The disparity map generation submodule includes:

[0046] A preprocessing unit, used for performing Sobel processing on the acquired image to obtain a gradient value;

[0047] The matching cost calculation unit is used for the matching cost calculation steps including: performing BT processing on the gradient value to obtain the first generation value, performing BT processing on the acquired image to obtain the second generation value; performing Census transformation processing on the acquired image to obtain the third generation value;

[0048] The target cost value calculation unit is used to perform weighted addition of the first generation value, the second generation value, and the third generation value to obtain the target cost value.

[0049] In one embodiment, the target cost value calculation unit is used to use the formula C=αC SobelX +β*C BT +γ*C census Calculate the target cost function, where C is the target cost value, C SobelX is the first generation value, C BT is the second generation value, C census It is the third generation value, α, β, and γ are all weight coefficients, and their value range is 0 to 1.

[0050] In one implementation, the non-occluded area is a mismatched area, and the parallax filling module includes:

[0051] A linear interpolation calculation submodule is used to use the invalid point in the mismatching area as a starting point, and when valid points with the smallest distance are queried in at least two directions, to perform linear interpolation using the disparity values ​​corresponding to the queried multiple valid points;

[0052] The second disparity filling submodule is used to perform disparity filling on invalid points in the mismatching area using the interpolation result.

[0053] In one embodiment, it further includes:

[0054] A median calculation submodule is used to use the disparity values ​​of valid points around the invalid point in the mismatching area to perform median calculation when no valid point with the smallest distance is found in at least two directions with the invalid point in the mismatching area as the starting point;

[0055] The third disparity filling submodule is used to perform disparity filling on invalid points in the mismatching area using the median calculation result.

[0056] In a third aspect, an electronic device is provided, including:

[0057] at least one processor; and a memory communicatively coupled to the at least one processor;

[0058] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute any of the above methods.

[0059] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute any of the above methods.

[0060] One embodiment of the above application has the following advantages or beneficial effects: by selecting an invalid point in the first occlusion area, and on the straight line passing through the invalid point, for both sides of the invalid point, respectively querying the first valid point and the second valid point with the smallest distance from the invalid point, and the corresponding first disparity value and second disparity value; when the absolute value of the difference between the first disparity value and the second disparity value is greater than the preset value, all the pixels between the first valid point and the second valid point belong to the second occlusion area. Since the occlusion area is usually a connected area and not a number of discrete points, the absolute value of the difference between the first disparity value and the second disparity value is greater than the preset value, indicating that the two valid points belong to different objects, or belong to the background and the foreground respectively, and it can be considered that this area is all caused by occlusion. Therefore, the invalid points caused by mismatching are excluded in the first occlusion area, and the occlusion area and the mismatching area are more accurately distinguished, so that the invalid points are filled more effectively, and the disparity filling effect is better, thereby improving the stereo matching effect of the image.

[0061] Other effects of the above optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0063] Figure 1 is a schematic diagram of a stereo matching method of an image provided according to an embodiment of the present application;

[0064] Figure 2 is a schematic diagram of a stereo matching method of an image provided according to another embodiment of the present application;

[0065] Figure 3 is a schematic diagram of a matching cost fusion method provided according to another embodiment of the present application;

[0066] Figure 4 A scene diagram of a stereo matching method for an image provided according to an embodiment of the present application;

[0067] Figure 5 The left and right figures are schematic diagrams of a Hamming distance calculation scenario provided according to another embodiment of the present application;

[0068] Figure 6 is a schematic diagram of a stereo matching device for images provided according to an embodiment of the present application;

[0069] Figure 7 is a schematic diagram of a stereo matching device for images provided according to another embodiment of the present application;

[0070] Figure 8is a schematic diagram of a stereo matching device for images provided according to an embodiment of the present application;

[0071] Fig. 9 It is a block diagram of an electronic device used to implement a stereo matching method of an image in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0073] In a specific embodiment, Figure 1 As shown, a stereo matching method for images is provided, comprising the following steps:

[0074] Step S110: Processing the acquired image to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area;

[0075] Step S120: selecting an invalid point in the first occlusion area, and querying the first valid point and the second valid point with the smallest distance from the invalid point on both sides of the straight line passing through the invalid point, and the corresponding first disparity value and second disparity value respectively;

[0076] Step S130: when the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset value, all pixel points between the first effective point and the second effective point belong to the second occlusion area;

[0077] Step S140: performing disparity filling on the second occlusion area and the non-occlusion area in the invalid point area to obtain a dense disparity map.

[0078] In one example, a binocular camera captures two viewpoint images of the same scene, i.e., a left image and a right image, and uses a stereo matching algorithm for preprocessing, matching cost calculation, matching cost fusion, disparity calculation, disparity optimization, and disparity filling, etc., to obtain a dense disparity map. First, the preprocessing can use the SobelX operator to process the left image and the right image to obtain two gradient values. Matching cost calculation: BT (Birchfield, Tomasi) processing is performed on the two gradient values ​​to obtain the first generation value of the left image and the right image matching. At the same time, BT (Birchfield, Tomasi) processing is directly performed on the left image and the right image to obtain the second generation value of the left image and the right image matching. Matching cost fusion: The first generation value and the second generation value are fused, and the weight coefficients of the fusion of the two can be 1 and 1 / 4. Of course, the matching cost calculation method includes but is not limited to the above method, and can also include other matching cost calculation methods, all of which are within the protection scope of this embodiment. In subsequent embodiments, another preferred matching cost calculation and fusion method will be provided. Disparity calculation: The (WTA, WinnerTake All) strategy can be used to calculate the disparity value of each pixel in the left image to obtain the left disparity map, and to calculate the disparity value of each pixel in the right image to obtain the right disparity map.

[0079] Since the disparity map still has some problems such as inaccurate disparity of occluded points, noise points, and mismatched points, the disparity map can be optimized by uniqueness detection, sub-pixel interpolation, left / right consistency detection, and connected area detection. It can include sub-pixel fitting, consistency check (Left / Right Consistency Check), uniqueness constraint (Uniqueness), remove small connected areas (Remove Peaks), median filter (Median Filter) and other operations to determine the invalid point area. Specifically, sub-pixel fitting improves the whole pixel accuracy to sub-pixel accuracy. The consistency check includes: 1. Obtain the left and right disparity maps; 2. For each pixel aaa in the left disparity map, calculate the pixel bbb of the same name point in the right disparity map; 3. Determine whether the absolute value of the difference between the disparity values ​​of pixel aaa and pixel bbb is less than a threshold (usually 1 pixel); 4. If it exceeds the threshold, the consistency check fails, and the disparity value of the corresponding position is changed to an invalid value, which belongs to the invalid point area. Unique constraints and elimination of small connected areas can further eliminate unreliable disparities and erroneous disparity areas that are connected into blocks; finally, through median filtering, noise points are further eliminated, and some small invalid areas are filled to eliminate some isolated outliers in the disparity map. After that, some points that are considered unreliable by disparity are marked as invalid pixels (invalid points for short) to obtain invalid point areas. Since the left image and the right image can be interchanged, only one of the images needs to be selected to mark the invalid point area. In some scenes, dense disparity maps or depth maps are required, that is, the disparity values ​​of all pixels need to be given, so it is necessary to fill the invalid points with disparity. Disparity filling is to assign a valid disparity value to the invalid points of the disparity map. The filled left and right disparity maps can be converted into depth maps. The application range of depth maps is very wide. Since they can record the distance of objects in the scene from the camera, they can be used for measurement, three-dimensional reconstruction, and synthesis of virtual viewpoints.

[0080] The invalid point area can be divided into an occlusion area and a mismatch area. The occlusion area is a pixel area that is visible on the left image but invisible on the right image due to foreground occlusion. For example, for the left image, the occlusion point generally exists at the position where the background area and the foreground area touch each other. The occlusion occurs because the foreground has a larger offset than the background, thereby covering the background. Preliminary determination of the first occlusion area: For example, for an invalid point on the left image, if no corresponding point can be found on the right image within all parallax ranges, the point is considered to belong to the first occlusion area. However, due to mismatching, some invalid points in the first occlusion area in the left image can also find corresponding pixel points on the right image. Therefore, the first occlusion area may contain invalid points that are mismatched. In order to further exclude invalid points of mismatch in the first occlusion area and divide the occlusion area more accurately, in this embodiment, an invalid point is selected in the first occlusion area, and on the straight lines in various directions passing through the invalid point, for example, the horizontal line, the vertical line, the straight line with an angle of 45° with the horizontal direction, the straight line with an angle of 60° with the horizontal direction, etc., if the first valid point is queried on the straight lines on both sides of the invalid point, which are the first valid point and the second valid point, respectively, and the corresponding disparity values ​​are the first disparity value and the second disparity value, then, when the absolute value of the difference between the first disparity value and the second disparity value is greater than the preset value, all the pixels between the first valid point and the second valid point belong to the second occlusion area. Similarly, the invalid points belonging to the second occlusion area are collected and the second occlusion area is divided out. Finally, the second occlusion area is excluded from the invalid point area, and the remaining part is the mismatch area, that is, the non-occlusion area. The second occlusion area and the non-occlusion area are filled with disparity to obtain a dense disparity map, so that the left disparity map and the right disparity map match more accurately.

[0081] In this embodiment, a stereo matching method for an image is provided, by selecting an invalid point in a first occlusion area, and querying the first valid point and the second valid point with the smallest distance from the invalid point, and the corresponding first disparity value and second disparity value on both sides of the invalid point on a straight line passing through the invalid point; when the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset value, all the pixels between the first valid point and the second valid point belong to the second occlusion area. Since the occlusion area is usually a connected area and not a number of discrete points, the absolute value of the difference between the first disparity value and the second disparity value is greater than the preset value, indicating that the two valid points belong to different objects, or belong to the background and the foreground respectively, and it can be considered that this area is all caused by occlusion. Therefore, the invalid points caused by mismatching are excluded in the first occlusion area, and the occlusion area and the mismatching area are distinguished more accurately, so that the invalid points are filled more effectively, and the disparity filling effect is better, thereby improving the stereo matching degree of the image.

[0082] In one embodiment, Figure 2 As shown, step S120 includes:

[0083] Step S121: When the straight line passing through the invalid point is a horizontal line, query the first valid point and the second valid point with the smallest distance from the invalid point for the left and right sides of the invalid point, and the corresponding first disparity value and second disparity value, respectively.

[0084] In one example, in the left disparity map or the right disparity map, starting from the selected invalid point, the first valid disparity point is found along the horizontal line to the left and right, respectively. These two points are recorded as p l and p r , and the corresponding disparity values ​​are recorded as d l and d r , if |d l -d r |>Th1, Th1 is the first preset value, then p l and p r All pixels between are considered to belong to the second occlusion area. If the left edge of the disparity map is found on the left or the right edge of the disparity map is found on the right, it is also considered that the pixels between p and p belong to the second occlusion area. r Go left or from p l All pixels on this horizontal line to the right belong to the second occlusion area.

[0085] In one embodiment, Figure 2 As shown, step S120 includes:

[0086] Step S122: When the straight line passing through the invalid point is a vertical line, query the first valid point and the second valid point with the smallest distance from the invalid point, and the corresponding first disparity value and second disparity value, respectively, for the upper side and the lower side of the invalid point.

[0087] In one example, in the left disparity map or the right disparity map, starting from the selected invalid point, the first valid disparity point is found along the vertical line upward and downward, respectively, and these two points are respectively recorded as p t and p b , and the corresponding disparity values ​​are recorded as d t and d b , if |d t -d b |>Th2, Th2 is the second preset value, then p t and p b All pixels between are considered to belong to the second occlusion area. If the upper boundary of the disparity map is found upward or the lower boundary is found downward, it is also considered that the pixels between p and p belong to the second occlusion area. b Up or from p t All pixels on this vertical line downward belong to the second occlusion area.

[0088] In this embodiment, since the viewing angles of the left and right images are different, the left and right cameras may cause occlusion of the upper and lower boundaries, so querying the valid points on the vertical line can determine the occlusion area on the upper and lower boundaries. In addition, the parallax values ​​of invalid points queried on the straight lines in other directions can be decomposed into the horizontal and vertical directions, and the comparison method of the horizontal and vertical directions is performed on the horizontal and vertical lines to determine whether all the pixel points between two valid points in other directions belong to the second occlusion area.

[0089] In one embodiment, Figure 2 As shown, step 140 includes:

[0090] Step 141: for multiple straight lines passing through the invalid point in the second occlusion area, respectively search for valid points with the smallest distances and corresponding disparity values ​​on the multiple straight lines, and the angles between the multiple straight lines are the same;

[0091] Step 142: Select a minimum disparity value from the multiple disparity values ​​queried, and use the minimum disparity value to perform disparity filling on invalid points in the second occlusion area.

[0092] In one example, the occlusion area is located in the discontinuous parallax area, one side is the foreground, the disparity value is large, and the other side is the background, the disparity value is small. When filling, it should be closer to the background pixel disparity, and the difference with the foreground disparity is large, so when filling, the surrounding background pixels should be selected as much as possible, and the foreground pixels should be avoided. For example, an invalid point is selected in the second occlusion area, and the invalid point is used as the starting point. 8 rays are emitted outward at angles such as upward, downward, left, right, upper left, upper right, lower left, and lower right (the angle between the 8 rays is 45°). The first valid point found in these 8 directions, that is, the first valid point to the eighth valid point, is selected. The minimum disparity value among these 8 valid points is selected as the disparity value of the invalid point. Of course, 5 straight lines passing through the invalid point can also be selected, and the angles between the 5 straight lines are the same, which are all within the protection scope of this embodiment.

[0093] In one embodiment, Figure 2 As shown, step 110 includes:

[0094] Step 111: preprocessing the acquired image, performing matching cost calculation, matching cost fusion, and disparity calculation to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area;

[0095] like Figure 3 As shown, step 111: the preprocessing step includes: performing Sobel processing on the acquired image to obtain a gradient value;

[0096] Step 1112: The matching cost calculation step includes: performing BT processing on the gradient value to obtain the first generation value, performing BT processing on the acquired image to obtain the second generation value; performing Census transformation processing on the acquired image to obtain the third generation value;

[0097] Step 1113: The step of matching cost fusion includes: weighted addition of the first generation value, the second generation value, and the third generation value to obtain the target cost value.

[0098] In one example, Figure 4 As shown in the figure, the left and right images are processed by SobelX to obtain the first gradient value and the second gradient value; the first gradient value and the second gradient value are processed by BT (Birchfield, Tomasi) to obtain the first generation value. The BT (Birchfield, Tomasi) processing is performed from the left image to the right image to obtain the second generation value. The Census transformation processing is performed from the left image to the right image to obtain the third generation value. Among them, the cost calculation method of Census is as follows: Figure 5 As shown in Figure 1, the Census transform uses the local grayscale difference in the pixel neighborhood to convert the pixel grayscale into a bit string. The grayscale value of the pixel in the neighborhood window (the window size is n×m, n and m are both odd numbers) is compared with the grayscale value of the center pixel of the window (the center pixel coordinates of the left figure are (u, v), and the center pixel coordinates of the right figure are (ud, v)). The Boolean value obtained by the comparison is mapped to a bit string, and finally the value of the bit string is used as the Census transform value C of the center pixel. s (u,v):

[0099]

[0100] Where n' and m' are the largest integers not greater than half of n and m, respectively. It is a bit-by-bit connection operation. The ξ operation is defined as follows: (u, v) is the coordinate of the center pixel of the left image, I represents the gray value, and i and j are loop variables.

[0101]

[0102] The matching cost calculation method based on Census transformation is to calculate the Hamming distance of the Census transformation values ​​of the two center pixels corresponding to the left and right images to obtain the third-generation value:

[0103] C(u,v,d):=Hamming(C sl (u,v),C sr (ud,v)) (2)

[0104] Among them, d is the number of pixels that differ between the corresponding horizontal coordinates of the left image and the right image, C sl (u, v) is the Census transformation value of the center pixel of the left image, C sl (ud, v) is the Census transform value of the center pixel of the right image. Assuming that the center pixel coordinates of the left image are u=100, v=75, and the disparity value is 20, calculate the grayscale value of the pixel in the neighborhood window (the window size is n×m, n and m are both odd numbers) and compare it with the grayscale value of the center pixel. After comparison, the first row sequence is obtained (the grayscale value of the pixel in the neighborhood window is 0 if it is greater than the grayscale value of the center pixel, and the grayscale value of the pixel in the neighborhood window is 1 if it is less than the grayscale value of the center pixel). Similarly, the center pixel coordinates of the right image are v=75, d=1, ud=99, and the disparity value is 40. The second row sequence is obtained in the same calculation method. Comparing these two sequences, if there are two differences, the calculated Hamming distance is 2.

[0105] In one embodiment, step 1113 includes:

[0106] C=αC SobelX +β*C BT +γ*C census

[0107] Among them, C is the target cost value, C SobelX is the first generation value, C BT is the second generation value, C census is the third generation value, α, β, and γ are all weight coefficients, and the value range is 0 to 1. For example, α is 1, β is 0, and γ is 0.5. Of course, the three weight coefficients can also take other values, without limitation, and are all within the protection scope of this embodiment. Traverse all the pixels in the left and right images, and calculate the target cost value of each pixel. If a certain pixel in the left image finds the minimum target cost value within the disparity range (for example, 1-100) in the right image, it is used as the disparity value of this pixel. Similarly, traverse all the pixels in the left image and calculate the disparity value corresponding to each pixel. Similarly, if a certain pixel in the right image finds the minimum target cost value within the disparity range (for example, 1-100) in the left image, it is used as the disparity value of this pixel. Similarly, traverse all the pixels in the right image and calculate the disparity value corresponding to each pixel.

[0108] In this embodiment, SobelX+BT and original image BT cost fusion are both solutions for stereo matching based on color information. On this basis, combined with Census transformation processing, that is, adding texture information, stereo matching can be made more accurate and the robustness can be effectively improved.

[0109] In one embodiment, if Figure 2 As shown, the non-blocked area is a mismatched area, and step S140 includes:

[0110] Step S143: taking the invalid point in the mismatched area as the starting point, and when valid points with the smallest distance are found in at least two directions, linear interpolation is performed using the disparity values ​​corresponding to the multiple found valid points;

[0111] Step S144: using the interpolation result to perform parallax filling on invalid points in the mismatched area.

[0112] In one example, the mismatch area is not in the occlusion area, and the neighboring pixels are all located on a continuous disparity surface. The disparity is continuous, so all pixels in the neighborhood can be considered when filling. For example, if valid points can be found on both the left and right sides of an invalid point in the mismatch area, then the disparity values ​​of the two valid points on the left and right are linearly interpolated, and the interpolation result is used as the disparity value of the invalid point. Of course, the method of selecting valid points of the invalid point's neighbors includes but is not limited to the above method, and linear interpolation can also be performed on the upper and lower valid points, the upper left and lower right valid points, the upper right and lower left valid points, etc., all within the protection scope of this embodiment.

[0113] In one embodiment, if Figure 2 As shown, step S140 includes:

[0114] Step S145: Taking the invalid point in the mismatching area as the starting point, if no valid point with the shortest distance is found along at least two directions, median calculation is performed using the disparity values ​​of the valid points around the invalid point in the mismatching area;

[0115] Step S146: using the median calculation result to perform parallax filling on invalid points in the mismatched area.

[0116] In one example, if no valid points can be found on the left and right sides of the invalid point in the mismatched area, the median calculation can be performed using the disparity values ​​of the valid points found in 8 directions around the invalid point, including up, down, left, right, upper left, upper right, lower left, and lower right. For example, 8 valid points are found in 8 directions, and the disparity values ​​of these 8 valid points are arranged in order of size to form a sequence. The variable value in the middle of the variable sequence is called the median, and the median is used as the median calculation result, and the median calculation result is used as the disparity value of the invalid point.

[0117] In another specific embodiment, Figure 6 As shown, a stereo matching device for images is provided, comprising:

[0118] An image processing module 110 is used to process the acquired image to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area;

[0119] The valid point query module 120 is used to select an invalid point in the first occlusion area, and query the first valid point and the second valid point with the smallest distance from the invalid point on both sides of the straight line passing through the invalid point, and the corresponding first disparity value and the second disparity value respectively;

[0120] The occlusion area determination module 130 is used for determining that all pixel points between the first effective point and the second effective point belong to the second occlusion area when the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset value;

[0121] The disparity filling module 140 is used to perform disparity filling on the second occlusion area and the non-occlusion area in the invalid point area to obtain a dense disparity map.

[0122] In one embodiment, Figure 7 As shown, the effective point query module 120 includes:

[0123] The first query submodule 121 is used to query the first and second valid points with the smallest distance from the invalid point, and the corresponding first and second disparity values, respectively, on the left and right sides of the invalid point when the straight line passing through the invalid point is a horizontal line.

[0124] In one embodiment, Figure 7 As shown, the effective point query module 120 includes:

[0125] The second query submodule 122 is used to query the first and second valid points with the smallest distance from the invalid point, and the corresponding first and second disparity values, respectively, on the upper and lower sides of the invalid point when the straight line passing through the invalid point is a vertical line.

[0126] In one embodiment, Figure 7 As shown, the parallax filling module 140 includes:

[0127] The third query submodule 141 is used to query the valid points with the shortest distance and the corresponding disparity values ​​on the multiple straight lines passing through the invalid point in the second occlusion area, and the angles between the multiple straight lines are the same;

[0128] The first filling submodule 142 is used to select a minimum disparity value from the queried multiple disparity values, and use the minimum disparity value to perform disparity filling on the invalid points in the second occlusion area.

[0129] In one embodiment, Figure 7 As shown, the image processing module 110 includes:

[0130] The disparity map generation submodule 111 is used to pre-process the acquired image, perform matching cost calculation, matching cost fusion, and disparity calculation to obtain a disparity map;

[0131] like Figure 8 As shown, the disparity map generation submodule 111 includes:

[0132] A preprocessing unit 1111 is used to perform Sobel processing on the acquired image to obtain a gradient value;

[0133] The matching cost calculation unit 1112 is used for the matching cost calculation steps including: performing BT processing on the gradient value to obtain the first generation value, performing BT processing on the acquired image to obtain the second generation value; performing Census transformation processing on the acquired image to obtain the third generation value;

[0134] The target cost value calculation unit 1113 is used to perform weighted addition on the first generation value, the second generation value, and the third generation value to obtain the target cost value.

[0135] In one embodiment, the target cost value calculation unit is used to use the formula C=αC SobelX +β*C BT +γ*C census Calculate the target cost function, where C is the target cost value, C SobelX is the first generation value, C BT is the second generation value, C census It is the third generation value, α, β, and γ are all weight coefficients, and their value range is 0 to 1.

[0136] In one embodiment, Figure 7 As shown, the non-occluded area is a mismatched area, and the parallax filling module 140 includes:

[0137] The linear interpolation calculation submodule 143 is used to use the invalid point in the mismatching area as the starting point, and when valid points with the smallest distance are found in at least two directions, to perform linear interpolation using the disparity values ​​corresponding to the multiple valid points found;

[0138] The second disparity filling submodule 144 is used to perform disparity filling on invalid points in the mismatching area using the interpolation result.

[0139] In one embodiment, it further includes:

[0140] The median calculation submodule 145 is used to use the disparity values ​​of the valid points around the invalid point in the mismatching area to perform median calculation when no valid point with the minimum distance is found in at least two directions with the invalid point in the mismatching area as the starting point;

[0141] The third disparity filling submodule 146 is used to perform disparity filling on invalid points in the mismatching area using the median calculation result.

[0142] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and will not be repeated here.

[0143] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0144] like Fig. 9 , is a block diagram of an electronic device according to a stereo matching method of an image according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0145] like Fig. 9 As shown, the electronic device includes: one or more processors 901, a memory 902, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 901 is taken as an example.

[0146] The memory 902 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform a stereo matching method for an image provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform a stereo matching method for an image provided in the present application.

[0147] The memory 902 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to a stereo matching method of an image in an embodiment of the present application (for example, the attached Figure 6 The processor 901 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 902, that is, implements a stereo matching method of an image in the above method embodiment.

[0148] The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of an electronic device according to a stereo matching method of an image, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 902 may optionally include a memory remotely arranged relative to the processor 901, and these remote memories may be connected to the above-mentioned electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The electronic device may further include: an input device 903 and an output device 904. The processor 901, the memory 902, the input device 903 and the output device 904 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0150] The input device 903 can receive input digital or character information, and generate key signal input related to the user settings and function control of the above electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 904 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0151] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0155] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0157] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A stereo matching method for images, characterized in that: include: Processing the acquired image to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area; Select an invalid point in the first occlusion area, and query a first valid point and a second valid point with the smallest distance from the invalid point on both sides of the invalid point on a straight line passing through the invalid point, and the corresponding first disparity value and second disparity value respectively; When the absolute value of the difference between the first disparity value and the second disparity value is greater than a preset value, all pixel points between the first valid point and the second valid point belong to the second occlusion area; Disparity filling is performed on the second occlusion area and the non-occlusion area in the invalid point area to obtain a dense disparity map.

2. The method according to claim 1, characterized in that The step of searching, on a straight line passing through the invalid point, for both sides of the invalid point, respectively for a first valid point and a second valid point with the smallest distance from the invalid point, comprises: When the straight line passing through the invalid point is a horizontal line, the first valid point and the second valid point with the smallest distance from the invalid point are queried for the left and right sides of the invalid point, respectively, as well as the corresponding first disparity value and second disparity value.

3. The method according to claim 1, characterized in that The step of searching, on a straight line passing through the invalid point, for both sides of the invalid point, respectively for a first valid point and a second valid point with the smallest distance from the invalid point, comprises: When the straight line passing through the invalid point is a vertical line, the first valid point and the second valid point having the smallest distance from the invalid point and the corresponding first disparity value and second disparity value are queried respectively for the upper side and the lower side of the invalid point.

4. The method according to claim 1, characterized in that: The performing parallax filling on the second occlusion area includes: For a plurality of straight lines passing through the invalid point in the second occlusion area, respectively querying valid points with the smallest distances and corresponding disparity values ​​on the plurality of straight lines, wherein the angles between the plurality of straight lines are the same; A minimum disparity value is selected from the multiple disparity values ​​queried, and the minimum disparity value is used to perform disparity filling on invalid points in the second occlusion area.

5. The method according to any one of claims 1 to 4, characterized in that: The step of processing the acquired image to obtain a disparity map includes: Preprocessing the acquired image, performing matching cost calculation, matching cost fusion, and disparity calculation to obtain the disparity map; Wherein, the preprocessing step includes: performing Sobel processing on the acquired image to obtain a gradient value; The matching cost calculation step includes: performing BT processing on the gradient value to obtain a first generation value, performing BT processing on the acquired image to obtain a second generation value; performing Census transformation processing on the acquired image to obtain a third generation value; The step of matching cost fusion includes: weighted addition of the first generation value, the second generation value, and the third generation value to obtain a target cost value.

6. The method according to claim 5, characterized in that The weighted addition of the first generation value, the second generation value and the third generation value to obtain a target cost value includes: in, cost value for the target, is the first generation value, is the second generation value, For the third generation of value, They are all weight coefficients, ranging from 0 to 1.

7. The method according to claim 1, characterized in that The non-blocking area is a mismatching area, and performing parallax filling on the non-blocking area in the invalid point area includes: Taking the invalid point in the mismatched area as a starting point, when valid points with the smallest distance are found in at least two directions, linear interpolation is performed using the disparity values ​​corresponding to the multiple found valid points; The interpolation result is used to perform disparity filling on invalid points in the mismatched area.

8. The method according to claim 7, characterized in that Also includes: Taking the invalid point in the mismatching area as the starting point, when no valid point with the shortest distance is found in at least two directions, median calculation is performed using the disparity values ​​of the valid points around the invalid point in the mismatching area; The invalid points in the mismatched area are filled with disparity using the median calculation result.

9. A stereo matching device for images, characterized in that: include: An image processing module, used to process the acquired image to obtain a disparity map and an invalid point area on the disparity map, wherein the invalid point area includes a first occlusion area; A valid point query module, configured to select an invalid point in the first occlusion area, and query a first valid point and a second valid point having the smallest distance from the invalid point on both sides of the invalid point on a straight line passing through the invalid point, and the corresponding first disparity value and second disparity value; an occlusion area determination module, configured to determine that, when an absolute value of a difference between the first disparity value and the second disparity value is greater than a preset value, all pixel points between the first valid point and the second valid point belong to a second occlusion area; The disparity filling module is used to perform disparity filling on the second occlusion area and the non-occlusion area in the invalid point area to obtain a dense disparity map.

10. The device according to claim 9, characterized in that The effective point query module comprises: The first query submodule is used to query the first valid point and the second valid point with the smallest distance from the invalid point, and the corresponding first disparity value and the second disparity value, respectively, on the left and right sides of the invalid point when the straight line passing through the invalid point is a horizontal line.

11. The device according to claim 9, characterized in that The effective point query module comprises: The second query submodule is used to query the first and second valid points with the smallest distance from the invalid point, and the corresponding first and second disparity values, respectively, on the upper and lower sides of the invalid point when the straight line passing through the invalid point is a vertical line.

12. The device according to claim 9, characterized in that The parallax filling module includes: A third query submodule is used to query, for a plurality of straight lines passing through the invalid point in the second occlusion area, respectively the valid points with the shortest distance and the corresponding disparity values ​​on the plurality of straight lines, wherein the angles between the plurality of straight lines are the same; The first filling submodule is used to select a minimum disparity value from the multiple disparity values ​​queried, and use the minimum disparity value to perform disparity filling on invalid points in the second occlusion area.

13. The device according to any one of claims 9 to 12, characterized in that: The image processing module comprises: A disparity map generation submodule is used to preprocess the acquired image, perform matching cost calculation, matching cost fusion, and disparity calculation to obtain the disparity map; The disparity map generation submodule includes: A preprocessing unit, used for performing Sobel processing on the acquired image to obtain a gradient value; A matching cost calculation unit, wherein the steps of matching cost calculation include: performing BT processing on the gradient value to obtain a first generation value, performing BT processing on the acquired image to obtain a second generation value; performing Census transformation processing on the acquired image to obtain a third generation value; The target cost value calculation unit is used to perform weighted addition on the first generation value, the second generation value, and the third generation value to obtain a target cost value.

14. The device according to claim 13, characterized in that The target cost value calculation unit is used to use the formula Calculate the target cost value, where cost value for the target, is the first generation value, is the second generation value, For the third generation of value, They are all weight coefficients, ranging from 0 to 1.

15. The device according to claim 9, characterized in that The non-blocked area is a mismatched area, and the parallax filling module includes: A linear interpolation calculation submodule, for taking the invalid point in the mismatching area as a starting point, and when valid points with the smallest distance are found in at least two directions, performing linear interpolation using the disparity values ​​corresponding to the multiple valid points found; The second disparity filling submodule is used to perform disparity filling on invalid points in the mismatching area using the interpolation result.

16. The device according to claim 15, characterized in that Also includes: A median calculation submodule, for taking the invalid point in the mismatching area as a starting point, and performing median calculation using the disparity values ​​of the valid points around the invalid point in the mismatching area when no valid point with the minimum distance is found along at least two directions; The third disparity filling submodule is used to perform disparity filling on invalid points in the mismatching area using the median calculation result.

17. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

Images