A method and device for generating standing tree parallax map
By using the median substitution and image segmentation technology of Census window in the generation of standing wood disparity maps, combined with Hamming distance and classification filling, the problem of mismatch of standing wood disparity maps in real scenes is solved, and the generation quality and anti-interference ability are improved. In complex standing wooden structure.
Patent Information
- Application Number
- CN202110871684.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-07-30
AI Technical Summary
When obtaining a wooden parallax map in real scenes, the existing technology is likely to have mismatch problems due to factors such as unstable lighting and similar texture structures. The deep learning methods lack prior knowledge and are difficult to apply to complex wooden structures.
Acquisition of stand-alone images through a binocular camera, use the median value of the Census window to replace the central pixel point, combine image segmentation and Hamming distance to obtain the initial matching generation value, determine the target parallax, and classify and fill for the invalid point types to optimize the parallax map generation.
It improves the generation quality of the standing wooden parallax map, reduces the mismatch rate, enhances the anti-interference ability, and can obtain fine parallax maps in complex backgrounds, and is suitable for weak textures and occluded areas.
Smart Images

Figure CN113554634B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating a standing tree disparity map. Background Art
[0002] Disparity map is the basis of depth map acquisition and 3D reconstruction. It is widely used in structure recovery of multi-visual images, surface data acquisition of 3D images, production of digital surface models and digital elevation models, etc. It is also widely used in aerospace, agriculture, forestry, mobile robots and other fields. Affected by factors such as unstable lighting and similar texture structures in real scenes, disparity maps are prone to mismatching problems, which seriously restricts their application scope. Therefore, improving the generation quality of disparity maps in real scenes is a hot topic in stereo matching research. The existing disparity map acquisition methods can be divided into traditional methods and deep learning methods.
[0003] The traditional method is to improve the quality of disparity maps by improving the classic stereo matching algorithm. In recent years, deep learning has developed rapidly and is favored by scholars at home and abroad. The use of convolutional neural networks (CNN) for stereo matching has achieved good results.
[0004] Images collected in real scenes are greatly affected by natural lighting, noise, etc., and generating disparity maps is relatively complex and difficult. The growth environment of standing trees is even more complex, affected by many factors such as lighting, occlusion, and texture, so obtaining standing tree disparity maps is challenging. Standing tree disparity maps contain standing tree objects and relative depth information, which can provide reference for standing tree factor measurement, three-dimensional reconstruction, etc. Traditional methods have a high mismatch rate, which restricts the scope of subsequent applications. Deep learning methods take a long time to learn through a lot of learning, and can obtain high-quality disparity maps, but lack a lot of prior knowledge in standing tree disparity maps, making training and learning methods unsuitable for complex standing tree structures. Summary of the invention
[0005] In view of the defects in the prior art, the present invention provides a method and device for generating a standing tree disparity map, so as to solve the problems that the prior method cannot handle practical applications and the standing tree growth environment is complex and the occlusion is serious.
[0006] In a first aspect, the present invention provides a method for generating a standing tree disparity map, comprising:
[0007] Acquire a standing tree image through a binocular camera, and calibrate and preprocess the standing tree image to obtain an image to be processed;
[0008] According to the image to be processed, a grayscale value replacement value and an initial matching cost value of a central pixel of a Census window are obtained;
[0009] In combination with the segmentation of the image to be processed, a total path cost value is obtained based on the initial matching cost value to determine the target disparity.
[0010] It can be seen from the above technical solution that the method for generating a standing tree disparity map provided by the present invention replaces the central pixel of the Census window by calculating the median of the eight neighborhoods, thereby avoiding the problem that the traditional Census window is too dependent on the central pixel. Since the traditional Census window takes values by comparing the size of the central pixel and the neighborhood pixel, it leads to large errors. When the texture is weak or the light is sufficient, part of the characteristic information of the standing tree is lost. The method of solving the median has a strong anti-interference ability. At the same time, the image contour obtained is clearer and can include more details.
[0011] Optionally, obtaining the grayscale value replacement value and the initial matching cost value of the central pixel point of the Census window includes:
[0012] Get the grayscale values of the eight neighboring pixels of the center pixel of the Census window;
[0013] Calculate the grayscale median of the eight neighborhood pixels of the central pixel of the Census window;
[0014] Replace the grayscale value of the central pixel of the Census window with the median grayscale value of the eight neighborhood pixels;
[0015] Get the initial matching cost value based on the Hamming distance.
[0016] Optionally, combining the to-be-processed image segmentation, obtaining a total path cost value based on the initial matching cost value, and determining the target disparity includes:
[0017] Performing clustering segmentation on the image to be processed so that the image to be processed is divided into a plurality of regions;
[0018] According to the divided areas, a single path aggregation cost value of a central pixel is obtained based on the initial matching cost value;
[0019] A total path cost value is obtained according to the single path aggregation cost value, and a target disparity is determined according to the total path cost value.
[0020] Optionally, the method further comprises:
[0021] forming a fitting curve based on the target disparity and associated disparities associated with the target disparity and determining a sub-pixel disparity value;
[0022] Acquire an invalid point in the disparity map according to the sub-pixel disparity value, and determine an invalid type of the invalid point based on a preset determination condition;
[0023] According to the invalid type, the invalid points are classified and filled with disparity values.
[0024] Optionally, the classifying and filling the invalid points according to the invalid type includes:
[0025] If the invalid point is a distorted point, taking the distorted point as the center, the disparity value of the distorted point is compared with the disparity values of the eight neighboring pixel points of the distorted point, and the disparity value of the distorted point is filled based on the disparity values of the eight neighboring pixel points;
[0026] If the disparity value of the distorted point is less than or equal to the minimum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the minimum disparity value;
[0027] If the disparity value of the distorted point is greater than or equal to the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the maximum disparity value;
[0028] If the disparity value of the distorted point is greater than the minimum disparity value among the eight neighborhood pixels and less than the maximum disparity value among the eight neighborhood pixels, the disparity value of the distorted point is filled with the average disparity value; wherein the average disparity value is determined by the minimum disparity value and the maximum disparity value.
[0029] Optionally, if the invalid point is a distorted point, the filling disparity value in, Y is the distortion point, M min is the minimum disparity value, M max is the maximum disparity value, M avg is the average disparity value, and Z is the set of disparity values of the eight neighborhoods of the distorted point.
[0030] Optionally, the classifying and filling the invalid points according to the invalid type further includes:
[0031] If the invalid point is a hole point, take the coordinate of the hole point Y as the starting point, traverse in the X direction and the Y direction respectively to find the first non-zero pixel point, form an adaptive window matrix, and determine the disparity filling value according to the elements in the adaptive window matrix.
[0032] Optionally, if the invalid point is a hole point, the hole point is used as the starting point, and the disparity values are traversed from the x direction and the y direction to determine the pixel points corresponding to the disparity value not being 0, to form an adaptive window matrix N, wherein: Y is the coordinate of the hole point, Y y Y is the coordinate of the first non-zero disparity pixel along the y-axis. x is the coordinate of the first non-zero disparity pixel point along the x direction of Y, (x′, y′) is the Y y The extension line parallel to the y direction and Y x The coordinates of the intersection of the extended lines parallel to the x-direction.
[0033] Optionally, based on the adaptive window matrix N, the filling disparity value in, is the sum of the disparity values of the elements in the adaptive window matrix N; is the number of non-zero elements in the adaptive window matrix N.
[0034] In a second aspect, the present invention provides a standing tree disparity map generating device, comprising:
[0035] An image acquisition module is used to acquire a standing tree image through a binocular camera, and calibrate and preprocess the standing tree image to obtain an image to be processed;
[0036] A calculation module, used for obtaining a grayscale value replacement value and an initial matching cost value of a central pixel point of a Census window according to the image to be processed;
[0037] The output module is used to obtain the total path cost value based on the initial matching cost value in combination with the segmentation of the image to be processed, and determine the target disparity.
[0038] Optionally, the calculation module is specifically used to:
[0039] Get the grayscale values of the eight neighboring pixels of the center pixel of the Census window;
[0040] Calculate the grayscale median of the eight neighborhood pixels of the central pixel of the Census window;
[0041] Replace the grayscale value of the central pixel of the Census window with the median grayscale value of the eight neighborhood pixels;
[0042] Get the initial matching cost value based on the Hamming distance.
[0043] Optionally, the output module is specifically used to:
[0044] Performing clustering segmentation on the image to be processed so that the image to be processed is divided into a plurality of regions;
[0045] According to the divided areas, a single path aggregation cost value of a central pixel is obtained based on the initial matching cost value;
[0046] A total path cost value is obtained according to the single path aggregation cost value, and a target disparity is determined according to the total path cost value.
[0047] Optionally, the device further includes an optimization module, specifically configured to:
[0048] forming a fitting curve based on the target disparity and associated disparities associated with the target disparity and determining a sub-pixel disparity value;
[0049] Acquire an invalid point in the disparity map according to the sub-pixel disparity value, and determine an invalid type of the invalid point based on a preset determination condition;
[0050] According to the invalid type, the invalid points are classified and filled with disparity values.
[0051] Optionally, the optimization module is further configured to:
[0052] If the invalid point is a distorted point, taking the distorted point as the center, the disparity value of the distorted point is compared with the disparity values of the eight neighboring pixel points of the distorted point, and the disparity value of the distorted point is filled based on the disparity values of the eight neighboring pixel points;
[0053] If the disparity value of the distorted point is less than or equal to the minimum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the minimum disparity value;
[0054] If the disparity value of the distorted point is greater than or equal to the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the maximum disparity value;
[0055] If the disparity value of the distorted point is greater than the minimum disparity value among the eight neighborhood pixels and less than the maximum disparity value among the eight neighborhood pixels, the disparity value of the distorted point is filled with the average disparity value; wherein the average disparity value is determined by the minimum disparity value and the maximum disparity value.
[0056] Optionally, the optimization module is further configured to:
[0057] If the invalid point is a distorted point, the fill disparity value in, Y is the distortion point, M min is the minimum disparity value, M max is the maximum disparity value, M avg is the average disparity value, and Z is the set of disparity values of the eight neighborhoods of the distorted point.
[0058] Optionally, the optimization module is further configured to:
[0059] If the invalid point is a hole point, take the coordinate of the hole point Y as the starting point, traverse in the X direction and the Y direction respectively to find the first non-zero pixel point, form an adaptive window matrix, and determine the disparity filling value according to the elements in the adaptive window matrix.
[0060] Optionally, the optimization module is further configured to:
[0061] If the invalid point is a hole point, take the hole point as the starting point, traverse the disparity values from the x direction and the y direction respectively to determine the pixel points corresponding to the disparity value not being 0, and form an adaptive window matrix N, where: Y is the coordinate of the hole point, Y y Y is the coordinate of the first non-zero disparity pixel along the y-axis. x is the coordinate of the first non-zero disparity pixel point along the x-axis direction of Y, (x′, y′) is the Y y The extension line parallel to the y-axis and Y x The coordinates of the intersection of the extended lines parallel to the x-axis.
[0062] Optionally, in the optimization module, based on the adaptive window matrix N, the filling disparity value in, is the sum of the disparity values of the elements in the adaptive window matrix N; is the number of non-zero elements in the adaptive window matrix N.
[0063] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.
[0064] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.
[0065] Based on the above technical solution, this application has the following beneficial effects:
[0066] 1) By calculating the median of the eight neighborhoods to replace the central pixel of the Census window, the problem of the traditional Census window being too dependent on the central pixel is avoided. Since the traditional Census window takes values by comparing the size of the central pixel and the neighborhood pixel, it leads to large errors. When the texture is weak or the light is sufficient, part of the characteristic information of the standing tree is lost. The method of solving the median has a strong anti-interference ability. At the same time, the image contour obtained is clearer and can include more details.
[0067] 2) This application uses an improved Census to perform cost calculation, and combines image segmentation for cost aggregation to obtain more effective information, and can still obtain a detailed standing tree disparity map under complex backgrounds. Using different penalty coefficients in several areas and in the same area can effectively reduce the mismatch rate in the cost aggregation process in weak texture and occluded areas.
[0068] 3) This application can fill invalid points in a targeted manner by targeting the invalid point type. Compared with the traditional SGM algorithm, it optimizes the invalid points in the disparity map, improves the image quality, and is universal in weak texture, occlusion and edge areas. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.
[0070] Figure 1 A flowchart of a method for acquiring a standing tree disparity map provided by an embodiment of the present invention is shown;
[0071] Figure 2A A schematic diagram of a standing tree image before correction provided by an embodiment of the present invention is shown;
[0072] Figure 2B A schematic diagram of a corrected standing tree image provided by an embodiment of the present invention is shown;
[0073] Figure 3 A schematic diagram of median replacement of a Census central pixel point provided by an embodiment of the present invention is shown;
[0074] Figure 4 A flowchart of a method for acquiring a standing tree disparity map provided by an embodiment of the present invention is shown;
[0075] Figure 5 A flowchart of a method for acquiring a standing tree disparity map provided by an embodiment of the present invention is shown;
[0076] Figure 6 A schematic diagram of 8-path aggregation provided by an embodiment of the present invention is shown;
[0077] Figure 7 A flowchart of a method for acquiring a standing tree disparity map provided by an embodiment of the present invention is shown;
[0078] Figure 8 A schematic diagram of a hole point filling model of an adaptive window matrix N provided by an embodiment of the present invention is shown;
[0079] Fig. 9 A comparison diagram of the effects of a standing tree disparity map acquisition method provided by an embodiment of the present invention and a GSM algorithm is shown;
[0080] Fig.10 A structural block diagram of a standing tree disparity map generating device provided by an embodiment of the present invention is shown;
[0081] Fig.11 A structural block diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0082] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.
[0083] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.
[0084] Standing tree disparity map is the basis of standing tree factor measurement and 3D reconstruction. Images collected in real scenes are greatly affected by natural lighting, noise, etc., and generating disparity maps is relatively complex and difficult. At present, the SGM algorithm is usually used for disparity map generation. This application is based on the SGM algorithm for improvement, thereby improving the generation effect of the disparity map and increasing the effective information for generating the disparity map.
[0085] See also Figure 1 , an embodiment of the present invention provides a method for obtaining a standing tree disparity map, comprising:
[0086] S101, acquiring a standing tree image through a binocular camera, and calibrating and preprocessing the standing tree image to obtain an image to be processed.
[0087] In this step, the standing tree images were captured using a portable USB3.0 binocular vision camera. A total of five different species of standing trees were collected, and for the convenience of research, the image resolution was uniformly set to 1280×720 pixels. The standing tree species photographed were garden trees, yucca, euonymus, hibiscus, etc., with a tree height of 1-3m. Due to different species, there are differences in height, so the shooting distance is 1.5-3m to ensure that the target standing tree is located in the center and intact. The collected standing tree images have the following characteristics: (1) Contains background standing trees, soil, grass, roads and other noise; (2) There are situations where the standing trees are blocked, the texture strength is different, and the height is inconsistent; (3) All were taken under non-uniform lighting on sunny days (with strong light interference).
[0088] At the same time, an essential step in stereo matching is binocular calibration. Calibration is to correct lens distortion by calculating intrinsic parameters (camera focal length fx, fy, optical center position coordinates and distortion coefficient) and extrinsic parameters (rotation R and translation T of the camera relative to the origin of the 3D world), see Figure 2A and Figure 2B , Figure 2A This is a schematic diagram of the standing tree image before correction. Figure 2BSchematic diagram of the corrected standing tree image. This embodiment adopts Zhang's classic chessboard calibration method, and selects a 12×9 grid, 3cm×3cm professional calibration plate to eliminate the influence of the calibration plate reflection and surface roughness on the calibration accuracy. In the specific implementation, the binocular camera is fixed in front of the calibration plate, and the calibration plate is always kept in the center of the image during the shooting process. In addition, the acquired standing tree image needs to be preprocessed for noise elimination and image color balancing.
[0089] S102: According to the image to be processed, a grayscale value replacement value and an initial matching cost value of a central pixel point of a Census window are obtained.
[0090] This step is different from the traditional Census algorithm. Instead, the eight neighboring pixels relative to the central pixel of the Census window are first determined, and then the grayscale values of the eight neighboring pixels are sorted, and the grayscale median of the eight neighboring pixels is determined according to the sorting; then the central pixel of the Census window is replaced with the obtained grayscale median.
[0091] S103 , combining the segmentation of the image to be processed, obtaining the total path cost value based on the initial matching cost value, and determining the target disparity.
[0092] Specifically, based on the mean shift algorithm, the standing tree image is segmented through repeated iterations and the standing tree image is divided into regions. In this step, a penalty coefficient μ is introduced to determine whether it is the same region. The obtained single path cost value is affected by the penalty coefficient μ, that is, the penalty coefficient is different in the same region or in several regions. Based on this, the standing tree image is cost aggregated, see Figure 6 , this step can adopt 8-path aggregation according to the image to be processed. Among all the disparity values generated according to the standing tree image, the disparity value corresponding to the minimum and minimum matching cost value determined by the WTA method is the target disparity, and the disparity map can be obtained according to the target disparity.
[0093] A method for generating a disparity map of standing trees in an embodiment of the present invention replaces the central pixel of the Census window by calculating the median of eight neighborhoods, thereby avoiding the problem that the traditional Census window is too dependent on the central pixel. Since the traditional Census window takes values by comparing the size of the central pixel and the neighborhood pixel, it leads to large errors. When the texture is weak or the light is sufficient, part of the characteristic information of the standing trees is lost. The method of solving the median has a strong anti-interference ability. At the same time, in step S103, different from the original SGM (Semi-Global Matching) algorithm to obtain the aggregation of single path cost values, a penalty coefficient is introduced to perform separate processing for the two situations where the pixel and the neighborhood pixel are in several areas and the same area.
[0094] Alternatively, see Figure 4 In order to obtain the cost value of the initial matching and thus determine the matching effect of the left image and the right image in the image to be processed, step S102 specifically further includes the following steps:
[0095] S1021. Obtain the grayscale values of the eight neighborhood pixels of the central pixel of the Census window.
[0096] S1022, calculating the grayscale median of the eight neighborhood pixels of the central pixel of the Census window.
[0097] S1023. Replace the grayscale value of the center pixel of the Census window with the median grayscale value of the eight neighborhood pixels.
[0098] In one possible implementation, see Figure 3 , sort the eight neighborhood pixels and record them as X1, X2, ..., X9, which does not include X5. X5 is used to mark the center pixel of the Census window. The specific grayscale value sorting is shown in Figure 2, which is arranged in ascending order as X1, X4, X3, X7, X9, X8, X2, and X6. Therefore, X7 and X9 are used as pixels for calculating the grayscale median, so the grayscale median X5 = (X7 + X9) / 2. When X5 is an integer, it is the replacement value of the center pixel. If X5 is not an integer, the value calculated by X5 is rounded to obtain the replacement value of the center pixel.
[0099] S1024. Obtain an initial matching cost value based on the Hamming distance.
[0100] Specifically, the Census transform maps the pixel point p and the window centered on it into a bit stream, and uses the bit stream as the Census transform code of the central pixel point. The mapping relationship of the traditional Census transform is as follows:
[0101]
[0102] In the above formula, I(p) and I(q) are the grayscale values of pixel points p and q. The corresponding bit stream is obtained from this mapping relationship, which is defined as follows:
[0103]
[0104] In the above formula, Str(x,y) is the Census transform code of the center pixel; (x,y) represents the coordinates of the center pixel; I(x,y) is the gray value of the center pixel; I(x+i,y+j) represents the gray value of the pixel with coordinates (x+i,y+j); l is half of the window width, and r is half of the window height; is a bit concatenation symbol. Based on the definition of the above formula, the Census transform code of the left and right images can be obtained. Based on the Census transform code, the matching cost is obtained by using the Hamming distance similarity measure, which is defined as follows:
[0105] C(x,y,d)=Hamming(str L (x,y),str R (xd,y))
[0106] In the above formula, C(x,y,d) represents the number of different bits in the two character bit strings of the left and right images, str L (x,y) is the Census transform code of the left image pixel (x,y), str L (xd,y) is the Census transform code of the left image and the right image with a difference of distance d. In the formula, C(x,y,d) is the initial matching cost, and the Hamming distance is to obtain str L (x,y) and str L (xd, y) is the sum of the number of corresponding bit differences between the two bit strings. The smaller the C(x, y, d) value is, the better the matching effect is.
[0107] The traditional Census algorithm is too dependent on the central pixel, is insensitive to illumination, and is greatly affected by noise. It only takes values based on the size of the adjacent pixels compared to the central pixel, resulting in large errors. Although it is more robust, it loses some standing tree detail information when the texture is weak and the illumination is sufficient. In order to obtain more detailed information, this embodiment replaces the grayscale value of the central pixel by obtaining the grayscale median of the eight neighborhoods of the central pixel, thereby improving the anti-interference ability, making the calculation result more accurate, and allowing the image to include more detailed information.
[0108] Alternatively, see Figure 5 In step S103, the following steps are also specifically included:
[0109] S1031 , clustering and segmenting the image to be processed so that the image to be processed is divided into a plurality of regions.
[0110] Specifically, the mean shift algorithm coarsens the complex background and then extracts the overall information. It achieves the segmentation of the standing tree image through repeated iterations, thereby achieving the division of several regions. It has the significant advantages of easy implementation, small amount of calculation, and accurate segmentation. By determining whether the color and texture areas of the standing trees are the same, this is used as the basis for standing tree segmentation, so the images segmented into the same area have the same color and texture. Based on the segmentation of color and texture, the segmented standing tree image retains more edge detail information.
[0111] It should be noted that, in this embodiment, the segmentation threshold is set to 30 based on the same segmentation standard.
[0112] S1032. Obtain a single path aggregation cost value of a central pixel point based on the divided area and the initial matching cost value.
[0113] Specifically, the current SGM algorithm uses the same penalty coefficient to obtain the energy function, so that the obtained energy function cannot include specific standing tree details, resulting in certain mismatches in weak texture or occluded areas. In this step, by distinguishing the same area and several areas, the penalty coefficient μ is introduced, so that the energy function is based on regional division and more detailed features are obtained.
[0114] Furthermore, the single-path aggregation cost is calculated by the following formula:
[0115]
[0116] Where p is a pixel, r is a single path, C(p,d) is the initial matching cost, and L is the single path aggregation cost. In the case of left and right paths, it is the neighboring pixels on the left side of p (aggregating from left to right) or on the right side (aggregating from right to left). L r (pr,d) is the aggregation cost value when the disparity of the previous pixel in the path is d, L r (pr,d-1) is the aggregation cost when the disparity of the previous pixel in the path is d-1, L r (pr,d+1) represents the aggregation cost when the disparity of the previous pixel in the path is d+1, min i L r (pr,i) represents the minimum value of all cost values of the previous pixel in the path; p1 and p2 are both penalty coefficients, which are applicable to pixels whose disparity values among the adjacent pixels of pixel p differ from the disparity value of p by 1, and p2 is applicable to pixels whose disparity values among the adjacent pixels of pixel p differ from the disparity value of p by more than 1.
[0117] S1033 . Obtain a total path cost value according to the single path aggregation cost value, and determine a target disparity according to the total path cost value.
[0118] like Figure 6 As shown, 8-path aggregation is used in this step. For a specific single path, refer to the following formula:
[0119] The total path cost S is calculated as follows:
[0120]
[0121] The disparity is calculated according to the Winner-Takes-All (WTA) algorithm, that is, among all the disparity values generated according to the standing tree images, the disparity value corresponding to the minimum matching cost is obtained, which is the target disparity.
[0122] By determining the energy function according to the region division, more detail information is retained and the matching rate of weak texture and occluded areas is improved during the cost aggregation process.
[0123] Optionally, after determining the target disparity, the obtained target disparity needs to be optimized, see Figure 7 , so the method also includes the following steps:
[0124] S201 , forming a fitting curve based on a target disparity and an associated disparity associated with the target disparity, and determining a sub-pixel disparity value.
[0125] Specifically, a fitting curve is determined according to the target disparity and the associated disparity associated with the target disparity map.
[0126] In a possible implementation, a parabola is fitted according to the target disparity d and its front disparity value d-1 and back disparity value d+1, and the lowest point of the parabola is used as the sub-pixel disparity value.
[0127] Since the winner-takes-all algorithm used in step S1033 is limited to calculating the pixel accuracy of the entire standing tree image and ignores the sub-pixel accuracy, in order to ensure that the standing tree disparity map is closer to the standing tree characteristics in the real scene, the disparity map accuracy is improved by parabola fitting in this step.
[0128] S202 , acquiring invalid points in the disparity map according to sub-pixel disparity values, and determining invalidity types of the invalid points based on preset determination conditions.
[0129] Specifically, in order to solve the problem of high mismatch rate due to occlusion of the standing tree disparity map, this embodiment uses the left-right consistency check to check invalid points. The preset judgment condition of the invalid point type is specifically the following formula:
[0130] D L (x,y)=D R (xD L (x,y),y)
[0131] If the above formula is not true and the parallax is not 0, it is determined to be a distortion point; otherwise, it is determined to be a hole point. L (x, y) represents the disparity value of a point in the left disparity map, D R (x,y) represents the disparity value of a point in the right disparity map.
[0132] S203: Classify the invalid points according to the invalid type and fill in the disparity value.
[0133] Optionally, the classifying and filling the invalid points according to the invalid type includes:
[0134] If the invalid point is a distorted point, taking the distorted point as the center, the disparity value of the distorted point is compared with the disparity values of the eight neighboring pixel points of the distorted point, and the disparity value of the distorted point is filled based on the disparity values of the eight neighboring pixel points;
[0135] If the disparity value of the distorted point is less than or equal to the minimum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the minimum disparity value;
[0136] If the disparity value of the distorted point is greater than or equal to the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the maximum disparity value;
[0137] If the disparity value of the distorted point is greater than the minimum disparity value among the eight neighborhood pixels and less than the maximum disparity value among the eight neighborhood pixels, the disparity value of the distorted point is filled with the average disparity value; wherein the average disparity value is determined by the minimum disparity value and the maximum disparity value.
[0138] Optionally, step S203 specifically includes:
[0139] If the invalid point is a distorted point, fill in the disparity value in, Y is the distortion point, M min is the minimum disparity value, M max is the maximum disparity value, M avg is the average disparity value, and Z is the set of disparity values of the eight neighborhoods of the distorted point.
[0140] Specifically, if the invalid point is determined to be a distorted point by step S202, if D L (y) <M min , then the distorted point disparity is filled with the minimum value; if M min <D L (y) <M max , the distorted point disparity is filled with the average value; if D L (y)>M max , the distorted point disparity is filled with the maximum value.
[0141] Optionally, the invalid points are classified and filled according to the invalid type, further comprising:
[0142] If the invalid point is a hole point, take the coordinate of the hole point Y as the starting point, traverse in the X direction and the Y direction respectively to find the first non-zero pixel point, form an adaptive window matrix, and determine the disparity filling value according to the elements in the adaptive window matrix.
[0143] Optionally, the invalid type of invalid points also includes hole points, so step S203 also includes filling the hole points, which is specifically as follows:
[0144] If the invalid point is a hole point, take the hole point as the starting point, traverse the disparity values from the x direction and the y direction respectively to determine the pixel points corresponding to the disparity value not being 0, and form an adaptive window matrix N, where: Y is the coordinate of the hole point, Y y is the coordinate of the first non-zero disparity pixel along the y direction, Y x is the coordinate of the first non-zero disparity pixel point along the x direction of Y, (x′, y′) is the Y y The extension line parallel to the y direction and Y x The coordinates of the intersection of the extended lines parallel to the x-direction.
[0145] Specifically, see Figure 8 , Figure 8 In this step, the hole point filling model corresponding to the adaptive window matrix N is traversed from the x direction and the y direction respectively, and the pixel points Y and Y y The distance is d y , pixel Y and Y x The distance is d x Based on this, the corresponding adaptive window matrix N is generated, and the pixel points that meet the hole point conditions in the disparity map are filled based on the adaptive window matrix N. y and Y to Y x In between, the number of pixels with a disparity value of 0 is not a fixed value according to the specific situation of generating the disparity map. Therefore, the matrix N realizes the adaptive characteristic, and the local adaptive matrix N is adaptively obtained to fill the disparity value of the hole points in the processed image.
[0146] Optionally, fill in the disparity value based on the adaptive window matrix N in, is the sum of the disparity values of the elements in the adaptive window matrix N; is the number of non-zero elements in the adaptive window matrix N.
[0147] Specifically, the sum S of the disparity values of all elements in the adaptive window matrix N is obtained, and the number n of non-zero elements in the matrix N is calculated. The average disparity value is calculated based on the above-obtained S and n, and the average disparity value is used to replace the disparity value of the hole point Y. Through the above method, the hole points are filled, the problem of low image quality caused by the existence of the hole points in the image is avoided, and the effect of smoothing the image is achieved.
[0148] See also Fig. 9 , Fig. 9This is a comparison chart of the effects of the SGM algorithm and the standing tree disparity map generation method provided by this application. The left column is the disparity map generated by the SGM algorithm, and the right column is the disparity map generated by this application. As can be seen from the figure, there are still too many invalid points in the disparity map generated by the traditional SGM algorithm. The invalid point filling method used in this application can fill the invalid points in a targeted manner according to the invalid point type, thereby improving the image quality, and is universal in weak texture, occlusion and edge areas, thereby improving the image quality.
[0149] In one embodiment, a standing tree disparity map generating device 30 is provided, see Fig.10 ,include:
[0150] The image acquisition module 301 is used to acquire the standing tree image through a binocular camera, and calibrate and preprocess the standing tree image to obtain an image to be processed;
[0151] A calculation module 302 is used to obtain a grayscale value replacement value and an initial matching cost value of a central pixel point of a Census window according to the image to be processed;
[0152] The output module 303 is used to obtain a total path cost value based on the initial matching cost value in combination with the segmentation of the image to be processed, and determine the target disparity.
[0153] Optionally, the calculation module is specifically used to:
[0154] Get the grayscale values of the eight neighboring pixels of the center pixel of the Census window;
[0155] Calculate the grayscale median of the eight neighborhood pixels of the central pixel of the Census window;
[0156] Replace the grayscale value of the central pixel of the Census window with the median grayscale value of the eight neighborhood pixels;
[0157] Get the initial matching cost value based on the Hamming distance.
[0158] Optionally, the output module is specifically used to:
[0159] Performing clustering segmentation on the image to be processed so that the image to be processed is divided into a plurality of regions;
[0160] According to the divided areas, a single path aggregation cost value of a central pixel is obtained based on the initial matching cost value;
[0161] A total path cost value is obtained according to the single path aggregation cost value, and a target disparity is determined according to the total path cost value.
[0162] Optionally, the device further includes an optimization module, specifically configured to:
[0163] forming a fitting curve based on the target disparity and the associated disparity of the associated target disparity and determining a sub-pixel disparity value;
[0164] Acquire an invalid point in the disparity map according to the sub-pixel disparity value, and determine an invalid type of the invalid point based on a preset determination condition;
[0165] According to the invalid type, invalid points are classified and filled with disparity values.
[0166] Optionally, the optimization module is further configured to:
[0167] If the invalid point is a distorted point, taking the distorted point as the center, the disparity value of the distorted point is compared with the disparity values of the eight neighboring pixel points of the distorted point, and the disparity value of the distorted point is filled based on the disparity values of the eight neighboring pixel points;
[0168] If the disparity value of the distorted point is less than or equal to the minimum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the minimum disparity value;
[0169] If the disparity value of the distorted point is greater than or equal to the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the maximum disparity value;
[0170] If the disparity value of the distorted point is greater than the minimum disparity value among the eight neighborhood pixels and less than the maximum disparity value among the eight neighborhood pixels, the disparity value of the distorted point is filled with the average disparity value; wherein the average disparity value is determined by the minimum disparity value and the maximum disparity value.
[0171] Optionally, the optimization module is further configured to:
[0172] If the invalid point is a distorted point, fill in the disparity value in, Y is the distortion point, M min is the minimum disparity value, M max is the maximum disparity value, M avg is the average disparity value, and Z is the set of disparity values of the eight neighborhoods of the distorted point.
[0173] Optionally, the optimization module is further configured to:
[0174] If the invalid point is a hole point, take the coordinate of the hole point Y as the starting point, traverse in the X direction and the Y direction respectively to find the first non-zero pixel point, form an adaptive window matrix, and determine the disparity filling value according to the elements in the adaptive window matrix.
[0175] Optionally, the optimization module is further configured to:
[0176] If the invalid point is a hole point, take the coordinate of the hole point Y as the starting point, traverse the disparity values from the x direction and the y direction respectively to determine the pixel points corresponding to the disparity value not being 0, and form an adaptive window matrix N, where: Y is the coordinate of the hole point, Y y Y is the coordinate of the first non-zero disparity pixel along the y-axis. x is the coordinate of the first non-zero disparity pixel point along the x-axis direction of Y, (x′, y′) is the Y y The extension line parallel to the y-axis and Y x The coordinates of the intersection of the extended lines parallel to the x-axis.
[0177] Optionally, in the optimization module, based on the adaptive window matrix N, the disparity value is filled in, is the sum of the disparity values of the elements in the adaptive window matrix N; is the number of non-zero elements in the adaptive window matrix N.
[0178] The standing tree disparity map generating device 30 provided in the embodiment of the present application adopts the same inventive concept as the above-mentioned standing tree disparity map generating method, and can achieve the same beneficial effects, which will not be described in detail here.
[0179] Based on the same inventive concept as the above-mentioned method for generating a standing tree disparity map, the embodiment of the present application further provides an electronic device 40, such as Fig.11 As shown, the electronic device 40 may include a processor 401 and a memory 402 .
[0180] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.
[0181] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0182] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; the above computer storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to: mobile storage devices, random access memory (RAM, Random Access Memory), magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)) and other media that can store program codes.
[0183] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: a mobile storage device, a random access memory (RAM, Random Access Memory), a magnetic storage device (such as a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical storage device (such as CD, DVD, BD, HVD, etc.), and a semiconductor memory (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), a solid-state drive (SSD)) and other various media that can store program codes.
[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A method for generating a standing tree disparity map, characterized in that: include: Acquire a standing tree image through a binocular camera, and calibrate and preprocess the standing tree image to obtain an image to be processed; According to the image to be processed, a grayscale value replacement value and an initial matching cost value of a central pixel of a Census window are obtained; Segmenting the image to be processed, obtaining a total path cost value based on the initial matching cost value, and determining a target disparity; The method further comprises: forming a fitting curve based on the target disparity and associated disparities associated with the target disparity and determining a sub-pixel disparity value; Acquire an invalid point in the disparity map according to the sub-pixel disparity value, and determine an invalid type of the invalid point based on a preset determination condition; According to the invalid type, the invalid points are classified and filled with disparity values, including: If the invalid point is a distorted point, taking the distorted point as the center, the disparity value of the distorted point is compared with the disparity values of the eight neighboring pixel points of the distorted point, and the disparity value of the distorted point is filled based on the disparity values of the eight neighboring pixel points; If the disparity value of the distorted point is less than or equal to the minimum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the minimum disparity value; If the disparity value of the distorted point is greater than or equal to the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the maximum disparity value; If the disparity value of the distorted point is greater than the minimum disparity value among the eight neighboring pixel points and less than the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with an average disparity value; wherein the average disparity value is determined by the minimum disparity value and the maximum disparity value; If the invalid point is a distorted point, the fill disparity value ;in, , Y is the distortion point, is the minimum disparity value, is the maximum disparity value, is the average disparity value, and Z is the disparity value set of the eight neighboring pixels of the distortion point; The classifying and filling the invalid points according to the invalid type further includes: If the invalid point is a hole point, taking the coordinate Y of the hole point as the starting point, traverse in the X direction and the Y direction respectively to find the first non-zero pixel point, form an adaptive window matrix, and determine the disparity filling value according to the elements in the adaptive window matrix; If the invalid point is a hole point, take the coordinate of the hole point Y as the starting point, traverse the disparity values from the x direction and the y direction respectively to determine the pixel points corresponding to the disparity value not being 0, and form an adaptive window matrix N, where: , Y is the coordinate of the hole point, is the coordinate of the first non-zero disparity pixel along the y-axis, is the coordinate of the first non-zero disparity pixel in the Y direction, for The extension line parallel to the y direction and The coordinates of the intersection of the extended lines parallel to the x-direction; Based on the adaptive window matrix N, the fill disparity value ;in, , is the sum of the disparity values of the elements in the adaptive window matrix N, and the pixel Y is The distance is , pixel Y and The distance is ; , is the number of non-zero elements in the adaptive window matrix N.
2. The method according to claim 1, characterized in that The step of obtaining the grayscale value replacement value and the initial matching cost value of the central pixel of the Census window includes: Get the grayscale values of the eight neighboring pixels of the center pixel of the Census window; Calculate the grayscale median of the eight neighborhood pixels of the central pixel of the Census window; Replace the grayscale value of the central pixel of the Census window with the median grayscale value of the eight neighborhood pixels; Get the initial matching cost value based on the Hamming distance.
3. The method according to claim 1, characterized in that The step of segmenting the image to be processed, obtaining a total path cost value based on the initial matching cost value, and determining a target disparity includes: Performing clustering segmentation on the image to be processed so that the image to be processed is divided into a plurality of regions; According to the divided areas, a single path aggregation cost value of a central pixel is obtained based on the initial matching cost value; A total path cost value is obtained according to the single path aggregation cost value, and a target disparity is determined according to the total path cost value.
4. A standing tree disparity map generating device, characterized in that: include: An image acquisition module is used to acquire a standing tree image through a binocular camera, and calibrate and preprocess the standing tree image to obtain an image to be processed; A calculation module, used for obtaining a grayscale value replacement value and an initial matching cost value of a central pixel point of a Census window according to the image to be processed; An output module, configured to segment the image to be processed, obtain a total path cost value based on the initial matching cost value, and determine a target disparity; The device also includes an optimization module, which is specifically used for: forming a fitting curve based on the target disparity and associated disparities associated with the target disparity and determining a sub-pixel disparity value; Acquire an invalid point in the disparity map according to the sub-pixel disparity value, and determine an invalid type of the invalid point based on a preset determination condition; Classify the invalid points and fill in disparity values according to the invalid type; The optimization module is further specifically used for: If the invalid point is a distorted point, taking the distorted point as the center, the disparity value of the distorted point is compared with the disparity values of the eight neighboring pixel points of the distorted point, and the disparity value of the distorted point is filled based on the disparity values of the eight neighboring pixel points; If the disparity value of the distorted point is less than or equal to the minimum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the minimum disparity value; If the disparity value of the distorted point is greater than or equal to the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with the maximum disparity value; If the disparity value of the distorted point is greater than the minimum disparity value among the eight neighboring pixel points and less than the maximum disparity value among the eight neighboring pixel points, fill the disparity value of the distorted point with an average disparity value; wherein the average disparity value is determined by the minimum disparity value and the maximum disparity value; The optimization module is further specifically used for: If the invalid point is a distorted point, the fill disparity value ;in, , Y is the distortion point, is the minimum disparity value, is the maximum disparity value, is the average disparity value, and Z is the set of disparity values of the eight neighborhoods of the distorted point; The optimization module is further specifically used for: If the invalid point is a hole point, taking the coordinate Y of the hole point as the starting point, traverse in the X direction and the Y direction respectively to find the first non-zero pixel point, form an adaptive window matrix, and determine the disparity filling value according to the elements in the adaptive window matrix; The optimization module is further specifically used for: If the invalid point is a hole point, take the hole point as the starting point, traverse the disparity values from the x direction and the y direction respectively to determine the pixel points corresponding to the disparity value not being 0, and form an adaptive window matrix N, where: , Y is the coordinate of the hole point, is the coordinate of the first non-zero disparity pixel along the y-axis, is the coordinate of the first non-zero disparity pixel along the x-axis. for The extension line parallel to the y-axis and The coordinates of the intersection of the extended lines parallel to the x-axis; In the optimization module, based on the adaptive window matrix N, the filling disparity value ; in, , is the sum of the disparity values of the elements in the adaptive window matrix N; , is the number of non-zero elements in the adaptive window matrix N.
