Binocular camera parallax calculation method and system based on illumination reference and electronic equipment

Abstract

The invention discloses a binocular camera parallax calculation method and system based on illumination reference and electronic equipment, and the method comprises the steps: obtaining a first left image and a first right image collected by a binocular camera under a first illumination condition, and calculating a corresponding first parallax image; marking a static target area in the first left image based on the first disparity map, and extracting an illumination reference feature of the static target area in a corresponding area image in the first left image; based on the illumination reference features, performing illumination correction on a second left image and a second right image collected by the binocular camera under a second illumination condition; based on the corrected second left image and the corrected second right image, a second disparity map is determined, and the second illumination condition is a weak illumination condition and / or a strong illumination condition. According to the technical scheme, the image / parallax value when illumination is normal is used as a reference, the complexity of a stereo matching method is reduced, and the accuracy of parallax calculation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of binocular vision, and in particular to a binocular camera parallax calculation method based on illumination reference, a binocular camera parallax calculation system based on illumination reference, and electronic equipment. Background Art

[0002] In the development of modern cities, smart cities have gradually become an important direction for urban construction. Smart cities can integrate a variety of advanced technologies with the help of cloud-edge-end architecture to realize the intelligence and efficiency of urban management services. In the framework of smart cities, the collection of urban information can be built with the help of smart street light systems. Smart street lights can not only undertake lighting functions, but also integrate a variety of sensors and communication equipment to become key nodes in smart cities. By installing binocular cameras on smart street lights, not only can image data of the field of view be collected in real time, but also the depth information of each target in the field of view can be obtained, thereby obtaining three-dimensional information of each target.

[0003] Lighting is the main factor that affects image formation. When the lighting conditions are poor (such as insufficient lighting, uneven lighting, or too strong lighting), the image pixel contrast obtained by the binocular camera is low, which not only makes the image difficult to distinguish, but also reduces the matching accuracy rate when performing stereo matching calculations because the pixel difference value is too small, thereby affecting the accuracy of disparity calculations. Therefore, it is necessary to improve the stereo matching method under poor lighting conditions, such as the Chinese invention patent application "A Local Stereo Matching Method Insensitive to Lighting" with publication number CN112150519A, to ensure that accurate disparity maps can be obtained, thereby improving the accuracy of smart city construction.

[0004] In the prior art, the stereo matching method of binocular cameras under poor lighting is usually more complicated, and is not related to the true value of the image / disparity map under normal lighting. It is only used as the true value for experimental verification, which requires repeated adjustment of the algorithm and prolongs the development cycle. Summary of the invention

[0005] The purpose of this application is to use the image / disparity value under normal lighting as a reference to reduce the complexity of the stereo matching method and improve the accuracy of the disparity calculation.

[0006] The technical solution of the first aspect of the present application is: a method for calculating the disparity of a binocular camera based on illumination reference is provided, the method comprising: step 1: obtaining a first left image and a first right image captured by a binocular camera under a first illumination condition, and calculating a corresponding first disparity map; step 2: based on the first disparity map, marking a static target area in the first left image, and extracting illumination reference features of the image of the corresponding area of ​​the static target area in the first left image; step 3: based on the illumination reference features, performing illumination correction on a second left image and a second right image captured by the binocular camera under a second illumination condition; step 4: determining a second disparity map based on the corrected second left image and the corrected second right image, wherein the second illumination condition is a weak illumination condition and / or a strong illumination condition.

[0007] In any of the above technical solutions, further, based on the first disparity map, the static target area in the first left image is marked, specifically including: Step 21: clustering the first disparity map to determine the clustering area; Step 22: when it is determined that the change in the pixel coordinates of any clustering area in the first disparity map of multiple consecutive frames is less than or equal to the static threshold, marking the clustering area as a static disparity area; Step 23: based on the pixel coordinates of the static disparity area, marking in the first left image, and recording the marked area as the static target area.

[0008] In any of the above technical solutions, further, in step 3, it specifically includes: step 31: based on the static target area, extracting the illumination features of the image at the corresponding position of the second left image to obtain the current illumination features; step 32: based on the illumination reference features and the current illumination features, calculating the regional incremental index corresponding to each static target area under the second illumination condition; step 33: calculating the mean of the regional incremental index, and using the mean to perform illumination correction on the second left image and the second right image.

[0009] In any of the above technical solutions, further, step 4 specifically includes:

[0010] Step 41: determining an edge image based on the corrected second left image and the corrected second right image;

[0011] Step 42: constructing an AD-Census cost value based on the edge image, the corrected second left image, and the corrected second right image;

[0012] Step 43: Based on the disparity value corresponding to the static target area, the AD-Census cost value is optimized, and based on the optimized AD-Census cost value, the fixed path aggregation cost value is determined, wherein the calculation formula of the optimized AD-Census cost value is:

[0013]

[0014] Where C(p, d) is the optimized AD-Census cost value, C0(p, d) is the AD-Census cost value, p is the pixel point in the corrected second left image, d is the disparity value of the pixel point p in the corrected second right image, and d′ is the static target area RtOR j The corresponding disparity value, λ is the optimization coefficient;

[0015] Step 44: construct a global energy function based on the fixed path aggregation cost value, and record the disparity value corresponding to the minimum value of the global energy function as the disparity value of the pixel point p;

[0016] Step 45: Traverse the pixel points in the corrected second left image to determine a disparity map.

[0017] In any of the above technical solutions, further, the edge image includes a left edge image and a right edge image, which are respectively determined by the corrected second left image and the corrected second right image. In step 41, the method for determining the left edge image specifically includes: converting the corrected second left image into a grayscale image, and filtering the grayscale image to generate a filtered image; calculating an adaptive threshold for each pixel in the filtered image, wherein the adaptive threshold is determined by the grayscale value of the pixel and the corresponding weight, and the weight is determined by the absolute value of the difference between the grayscale value of the pixel and the grayscale mean in the corresponding pixel window; based on the adaptive threshold, the Sobel edge detection algorithm is used to determine the left edge image.

[0018] In any of the above technical solutions, further, the grayscale image is filtered to generate a filtered image, which specifically includes: based on a preset filtering window, calculating the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; when it is determined that the difference between the minimum grayscale value and the maximum grayscale value is less than the grayscale threshold, the grayscale value of the current center point is recorded as the center point grayscale value; when it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, judging whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value, if so, recording the grayscale median as the center point grayscale value, if not, increasing the size of the preset filtering window according to the preset step size, and recalculating the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; based on the center point grayscale value, generating a filtered image.

[0019] In any of the above technical solutions, further, the edge image includes a left edge image and a right edge image, and step 42 specifically includes: converting the corrected second left image and the corrected second right image into grayscale images, which are respectively recorded as a left grayscale image and a right grayscale image; based on the left grayscale image and the right grayscale image, respectively, using a sliding window method, comparing the grayscale value of each pixel in the window with the grayscale value of the center pixel of the window, and generating a left grayscale 01 string and a right grayscale 01 string corresponding to the center pixel of the window; based on the left edge image and the right edge image, respectively, using a sliding window method, according to the distribution of pixel points in the window as edge points, generating a left edge 01 string and a right edge 01 string of the center pixel of the window; splicing the left grayscale 01 string and the right grayscale 01 string with the left edge 01 string and the right edge 01 string, respectively, to generate a left code and a right code of the center pixel of the window; calculating a Census value based on the Hamming distance of the left code and the right code, and constructing an AD-Census cost value based on the Census value, the corrected second left image and the corrected second right image.

[0020] In any of the above technical solutions, further, the calculation formula of the AD-Census cost value is:

[0021] C0(p,d)=ρ(AD,λ AD )+ρ(Census,λ census )

[0022]

[0023] In the formula, C0(p,d) is the AD-Census cost, ρ() is the mapping function, AD is the difference in the absolute value of the pixel, Census is the Census value, which indicates the similarity of the relative relationship between the adjacent areas of the pixel, and λ AD , census is the preset parameter, is the value of the q channel in the RGB three channels corresponding to the pixel point p in the corrected second left image, is the value of the q channel in the RGB three channels corresponding to the pixel point after the corresponding pixel point p in the corrected second right image is offset by the parallax d.

[0024] The technical solution of the second aspect of the present application is: a binocular camera disparity calculation system based on illumination reference is provided, the system comprising: a first disparity calculation unit, the first disparity calculation unit is configured to: obtain a first left image and a first right image captured by a binocular camera under a first illumination condition, and calculate a corresponding first disparity map; a feature calculation unit, the feature calculation unit is configured to: based on the first disparity map, mark the static target area in the first left image, and extract the illumination reference feature of the image of the corresponding area of ​​the static target area in the first left image; a correction unit, the correction unit is configured to: based on the illumination reference feature, perform illumination correction on the second left image and the second right image captured by the binocular camera under a second illumination condition; a second disparity calculation unit, the second disparity calculation unit is configured to: determine a second disparity map based on the corrected second left image and the corrected second right image, wherein the second illumination condition is a weak illumination condition and / or a strong illumination condition.

[0025] The technical solution of the third aspect of the present application is: an electronic device is provided, including: a memory, the memory stores execution instructions; a processor, the processor executes the execution instructions stored in the memory, so that the processor executes the binocular camera parallax calculation method based on illumination reference according to any one of the technical solutions of the first aspect above.

[0026] The beneficial effects of this application are:

[0027] The technical solution in this application is based on the premise that the field of view of the binocular camera installed on the smart lamp pole will not change. First, the static target area is determined in the disparity map under normal lighting conditions, and then the lighting reference features are extracted in the corresponding left image based on the coordinates of the area. Finally, the lighting reference features are used to correct the left and right images under abnormal lighting conditions to reduce the complexity of the stereo matching method and obtain an accurate disparity map.

[0028] In order to improve the accuracy of disparity calculation, the present application also proposes a method of using edge images to assist in calculating disparity maps. In the case of a disparity map of a feature area in a known static target area, by adding the difference between the calculated disparity and the known disparity in the area to the cost value, the calculated result can be brought closer to the known standard reference result, thereby improving its matching accuracy and thus improving the matching accuracy of the entire image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0030] Figure 1 is a schematic flow chart of a binocular camera parallax calculation method based on illumination reference according to an embodiment of the present application;

[0031] Figure 2 It is a schematic block diagram of a binocular camera disparity calculation system based on illumination reference according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0034] Embodiment 1:

[0035] In the scenario where the binocular camera is applied to smart street lights, for any calibrated binocular camera, since it is installed in a fixed position, its field of view is a fixed area, and in this area, there must be some static targets, such as trees, buildings, signboards, etc. Therefore, this embodiment proposes to use the lighting characteristics of these static targets under normal lighting conditions as lighting reference characteristics to correct the images obtained by the binocular camera under abnormal lighting conditions (weak lighting conditions and / or strong lighting conditions) to obtain an accurate disparity map, thereby improving the accuracy of the depth (distance) data of each target in the process of building a smart city.

[0036] This embodiment is now described by taking the change of illumination conditions caused by time change as an example.

[0037] like Figure 1 As shown, this embodiment provides a binocular camera parallax calculation method based on illumination reference, the method comprising:

[0038] Step 1: Obtain a first left image and a first right image captured by a binocular camera under a first illumination condition, and calculate a corresponding first disparity map;

[0039] Step 2: Based on the first disparity map, the static target area in the first left image is marked, and the illumination reference features of the image corresponding to the static target area in the first left image are extracted;

[0040] Step 3: Based on the illumination reference feature, illumination correction is performed on the second left image and the second right image captured by the binocular camera under the second illumination condition;

[0041] Step 4: Determine a second disparity map based on the corrected second left image and the corrected second right image, wherein the second lighting condition is a weak lighting condition and / or a strong lighting condition.

[0042] Specifically, 10 a.m. is set as a normal lighting condition, i.e., the first lighting condition; 17 p.m. is set as an abnormal lighting condition (weak lighting), i.e., the second lighting condition. Therefore, in this embodiment, the image (first left image, first right image) acquired by the binocular camera under the first lighting condition is used as a reference for lighting correction, and the image acquired under the second lighting condition is corrected.

[0043] For a binocular camera, its left and right cameras are installed on the same beam module. The first left image taken by the left camera can be selected as a reference. The first disparity map corresponding to the first left image can be calculated through the stereo matching method in binocular stereo vision. The specific calculation process will not be repeated here.

[0044] Considering that the binocular camera is installed on the smart street light, the targets within its field of view can be divided into static targets and dynamic targets (such as pedestrians and vehicles). For static targets, the distance (depth) between them and the binocular camera will remain unchanged, that is, the disparity corresponding to the static target in the disparity map is a fixed value, and the image position in the disparity map will remain unchanged.

[0045] Based on this situation, when the disparity of a certain area after clustering in the first disparity map of multiple consecutive frames is a certain value or changes very little, the area can be identified as a static target area. In addition, the first disparity map corresponds to the first left image, and the first left image can be annotated based on the static target area, and the annotated image target is the static target.

[0046] For the same target, the illumination characteristics reflected under different illumination conditions are different. Therefore, when performing illumination correction, the illumination characteristics corresponding to the static target under the first illumination condition can be used as a reference, and used as the illumination reference characteristics to correct the second left and right images under the second illumination condition. During correction, the illumination reference characteristics corresponding to each static target can be weighted, or the mean of the illumination reference characteristics can be calculated. Afterwards, the above calculation results can be used to correct the entire second left image and the second right image to obtain an accurate disparity map.

[0047] In any of the above embodiments, further, based on the first disparity map, marking the static target area in the first left image specifically includes:

[0048] Step 21: clustering the first disparity map to determine a clustering area;

[0049] Step 22: when it is determined that the change in pixel coordinates of any cluster area in the first disparity map of multiple consecutive frames is less than or equal to the static threshold, the cluster area is marked as a static disparity area;

[0050] Step 23: Based on the pixel coordinates of the static disparity area, mark it in the first left image, and record the marked area as the static target area.

[0051] Specifically, for the disparity map, the space and outer contour occupied by any target on it are continuous. Based on this continuity, a segmentation algorithm (such as clustering and connected domain analysis) can be used to cluster the first disparity map to determine the clustering area corresponding to each target within the field of view of the binocular camera, which is recorded as StO. i .

[0052] Each cluster area can introduce the concept of a target frame, and use the pixel coordinates of the upper left corner of its minimum bounding rectangle as its position coordinates (pixel coordinates) in the first disparity map, and the length and width of the minimum bounding rectangle as size information, which together constitute the target frame information (x, y, w, h) i .

[0053] When the position coordinates of a cluster area in the first disparity map of multiple consecutive frames obtained within a preset sampling time interval (such as the first disparity map obtained every 30 seconds from 9:30 to 10:30 every day for 5 consecutive days) do not change or the change is less than the set static threshold, the cluster area is marked as a static disparity area.

[0054] The disparity map corresponds to the left image. Therefore, the pixel coordinates of this static area can be used to mark it in the first left image and record it as the static target area StOR. j , the corresponding target box information is (x, y, w, h) j .

[0055] It should be noted that, in this embodiment, when the static target area in the first left image is annotated, the annotation is performed based on the first disparity map, rather than by performing target recognition / detection on the target in the first left image. This is due to the following reasons:

[0056] 1. When the lighting conditions are abnormal, if the network model is used to identify the target in the image obtained by the binocular camera, the recognition accuracy will be greatly reduced.

[0057] 2. If the target recognition model is deployed on the binocular camera, the production cost of the binocular camera will increase. However, considering that the binocular camera itself needs to perform parallax calculation, and clustering and area labeling based on parallax, the computing power consumed and the increased cost are much lower than the cost of deploying the model.

[0058] 3. If the target recognition model in the cloud is used for target recognition, and the recognition result / labeling result is then sent back to the binocular camera, the binocular camera needs to have not only the data sending function but also the data receiving function, which will also increase the production cost of the binocular camera.

[0059] In the acquired static target area StOR j , j = 1, 2, 3 ... m, corresponding to the first left image of multiple frames, for each static target area StOR j For example, a multi-scale fusion method can be used to extract the illumination reference features of the corresponding area image in the first left image. The process can be:

[0060] Firstly, multiple first left images are cropped based on the static target area, and the cropped images are subjected to three-layer Gaussian pyramid decomposition and Laplacian pyramid calculation: each layer of the image is blurred by a Gaussian filter and downsampled to obtain the next layer of the image; the Laplacian pyramid of each layer is calculated by subtracting the upsampled next layer of the Gaussian pyramid image from the current Gaussian pyramid layer.

[0061] Secondly, each layer of the pyramid image is fused, where the fusion rule is average fusion. Starting from the top layer, upsampling and addition operations are performed layer by layer to reconstruct the fused image, and finally the fused image is obtained.

[0062] Finally, the pre-trained convolutional neural network ResNet model is used as the illumination feature extraction network to extract the StOR of each static target area. j The corresponding fused image is subjected to feature extraction, and the extracted feature vector is the illumination reference feature F j , that is: F j =ResNet(StOP j ).

[0063] In any of the above embodiments, further, step 3 specifically includes:

[0064] Step 31: Based on the static target area, extract the illumination feature of the image at the corresponding position of the second left image to obtain the current illumination feature;

[0065] Step 32: Calculate the regional increment index corresponding to each static target area under the second illumination condition based on the illumination reference feature and the current illumination feature;

[0066] Step 33: Calculate the mean of the regional increment index, and use the mean to perform illumination correction on the second left image and the second right image.

[0067] Specifically, under the second illumination condition (such as night or cloudy day), the illumination condition of the binocular camera changes seriously, which affects the accuracy of parallax calculation. Therefore, it is necessary to use the illumination reference feature F of the image obtained under the first illumination. j , perform illumination correction on the currently acquired image to ensure the accuracy of parallax calculation.

[0068] According to the known static target area StOP j Target box information (x, y, w, h) j , intercept the second left image currently collected, and get StO j Image, corresponds to the image of the jth static object.

[0069] The captured image StO j Image is input into the ResNet model to extract the current lighting features Right now:

[0070]

[0071] Introducing the attention mechanism, the illumination reference feature F j Applied to the current illumination features corresponding to each static target area in the current second left image Get regional enhancement features Right now:

[0072]

[0073] The fused image is denoted as I ref , as the reference image, the image StO j The enhanced image is denoted as I enh , the loss function is defined as follows:

[0074]

[0075] In the formula, From image I ref 、Image I enh The expected value extracted from , D() is the discriminator. By minimizing the loss function, the enhanced image I enh As close as possible to image I in terms of lighting characteristics ref .

[0076] Use the optimization algorithm to train the model and obtain regional enhancement features by minimizing the loss function

[0077] Considering that the targets in different areas are different (materials, colors), resulting in different responses to illumination, the corresponding areas have different enhancement indexes. Therefore, for targets in different areas (such as roads, trees, buildings, etc.), the regional incremental index En after illumination enhancement is calculated for each area. j , the corresponding calculation formula is:

[0078]

[0079] Based on the incremental indicators of each region En j , calculate its mean Then, the second left image and the second right image ImageP are subjected to illumination correction to obtain the image EnImage used for parallax calculation:

[0080]

[0081] In the formula, Enhance() is the illumination correction formula, and the mean As an enhancement index, ImageP is enhanced to achieve illumination correction.

[0082] In any of the above embodiments, further, in order to improve the accuracy of disparity calculation, this embodiment also proposes a method for using edge images to assist in calculating a disparity map, and step 4 specifically includes:

[0083] Step 41: determining an edge image based on the corrected second left image and the corrected second right image;

[0084] Step 42: construct an AD-Census cost value based on the edge image, the corrected second left image and the corrected second right image; wherein the calculation formula of the AD-Census cost value is:

[0085] C0(p,d)=ρ(AD,λ AD )+ρ(Census,λ census )

[0086]

[0087] In the formula, C0(p,d) is the AD-Census cost, ρ() is the mapping function, AD is the difference in the absolute value of the pixel, Census is the Census value, which indicates the similarity of the relative relationship between the adjacent areas of the pixel, and λ AD , census is the preset parameter, is the value of the q channel in the RGB three channels corresponding to the pixel point p in the corrected second left image, is the value of the q channel in the RGB three channels corresponding to the pixel point after the corresponding pixel point p in the corrected second right image is offset by the parallax d.

[0088] Step 43: Based on the disparity value corresponding to the static target area, the AD-Census cost value is optimized, and based on the optimized AD-Census cost value, the fixed path aggregation cost value is determined, wherein the calculation formula of the optimized AD-Census cost value is:

[0089]

[0090] Where C(p, d) is the optimized AD-Census cost value, C0(p, d) is the AD-Census cost value, p is the pixel point in the corrected second left image, d is the disparity value of the pixel point p in the corrected second right image, and d′ is the static target area StOR j The corresponding disparity value, λ is the optimization coefficient;

[0091] Specifically, in common cost aggregation algorithms, the cost value of each pixel point often has the same weight, that is, the optimal matching solution is obtained by each point in each area with the same contribution. However, in the present embodiment, when the characteristic area disparity map in the static target area is known, by adding the difference between the calculated disparity and the known disparity in the area to the cost value, the calculation result can be brought closer to the known standard reference result, so as to improve its matching accuracy and thus improve the matching accuracy of the entire image. The calculation formula of the optimized AD-Census cost value is:

[0092]

[0093] Where C(p, d) is the optimized AD-Census cost value, C0(p, d) is the AD-Census cost value, p is the pixel point in the corrected second left image, d is the disparity value of the pixel point p in the corrected second right image, and d′ is the static target area RtOR j The corresponding disparity value, λ is the optimization coefficient.

[0094] Step 44: construct a global energy function based on the fixed path aggregation cost value, and record the disparity value corresponding to the minimum value of the global energy function as the disparity value of the pixel point p;

[0095] Step 45: Traverse the pixel points in the corrected second left image to determine a disparity map.

[0096] Specifically, when calculating the matching cost for pixel matching, a global matching algorithm is used. In order to reduce the system complexity and operation time, a unidirectional dynamic programming algorithm is introduced to aggregate the cost value based on a fixed path. The calculation formula is as follows:

[0097]

[0098] Among them, L r (p,d) represents the aggregation cost of pixel p along path r, P1 and P2 are smoothing penalty terms, It represents the minimum value of all aggregated cost values ​​of the previous pixel of p on path r, where the distance between i and d is greater than 1, that is, i is a value other than d, d+1, and d-1 within the range allowed by d.

[0099] The total cost of pixel P is C aggr (p,d)=∑ i L r (p, d), that is, summing up the multi-path costs. In this scheme, the 8-path method is selected.

[0100] The global energy function is defined as:

[0101]

[0102] Where E smo (d) is the smoothing term, which is penalized according to the distance between the two corresponding points in the left and right images.

[0103] Finally, a winner-takes-all algorithm can be used to obtain a disparity map based on the principle of minimum global energy function. After obtaining the disparity map, consistency detection, interpolation filling, etc. can be used as disparity optimization algorithms to optimize the disparity map.

[0104] In any of the above embodiments, further, the edge image includes a left edge image and a right edge image, which are respectively determined by the corrected second left image and the corrected second right image. In step 41, the method for determining the left edge image specifically includes:

[0105] The corrected second left image is converted into a grayscale image, and the grayscale image is filtered to generate a filtered image; wherein the filtering method can be Gaussian filtering, adaptive median filtering, and Gaussian smoothing can also be performed on the converted grayscale image.

[0106] Calculate the adaptive threshold of each pixel in the filtered image, where the adaptive threshold is determined by the gray value of the pixel and its corresponding weight, and the weight is calculated by the absolute value of the difference between the gray value of the pixel and the gray mean in the corresponding pixel window;

[0107] Based on the adaptive threshold, the Sobel edge detection algorithm is used to determine the left edge image.

[0108] Specifically, the calculation methods of the left edge image and the right edge image are the same. The calculation of the left edge image is taken as an example for explanation, and an edge detection algorithm based on a four-directional Sobel edge detection operator is adopted.

[0109] Taking into account the large difference in grayscale values ​​between the acquired images due to environmental influences, the color intensity of different areas of the same image is also different. The grayscale value of the overall pixel in the darker area is quite different from that in the brighter area, and the gradient value also has a difference in amplitude. Therefore, it is necessary to adaptively adjust the threshold according to the specific situation of the regional pixels.

[0110] Therefore, this embodiment proposes to calculate the weight of each pixel point participating in the mean calculation based on the distance between it and the mean. The larger the distance, the smaller its proportion, which can reflect the overall brightness level in the local area, has good adaptability to brightness changes, and suppresses interference from overly bright or dark areas. At the same time, it can effectively avoid the influence of individual points with large deviations in the window, so that the data can better reflect the level of most pixel points in the window and reduce the probability of false detection.

[0111] In the process of calculating the adaptive threshold of each pixel in the filtered image, firstly, in a 3×3 pixel window centered at any pixel i, the mean grayscale value of the 9 pixels in the window is calculated. Then calculate the mean The absolute value of the difference between the grayscale values ​​of these 9 pixels Among them, m is the label of these 9 pixels. If the 9 pixels in the 3×3 pixel window are sorted from left to right and from top to bottom, then when m=5, it corresponds to the pixel i.

[0112] According to the absolute value of the difference Calculate the weight w of pixel i i , the larger the difference, the smaller the weight, and the corresponding calculation formula is:

[0113]

[0114] Finally, with z i w i as an adaptive threshold.

[0115] Then, based on the Sobel edge detection algorithm, we can get the four gray value change gradients G0, G 45 , G 90 , G 135 , and then sum these four gray value gradients using the following formula to calculate the composite gradient amplitude G(x,y) and gradient direction θ(x,y) of the pixel:

[0116]

[0117] Finally, the Otsu (maximum inter-class variance method) algorithm is used to determine the edge points. During the determination process, each pixel is first checked along the gradient direction. If the gradient amplitude of this pixel is not the local maximum in this direction (compared with the adjacent pixels in this direction), then the gradient amplitude of this pixel will be suppressed, that is, set to zero or ignored.

[0118] Then scan each pixel in the filtered image, compare its gradient amplitude G(x, y) with the adaptive threshold, and record the pixel points greater than the adaptive threshold as strong boundaries, which are displayed as 1 in the binary image; the pixel points less than the adaptive threshold but greater than 1 / 2 of the adaptive threshold are recorded as weak boundaries. If the weak boundary is connected to the strong boundary within its 3×3 neighborhood, it is retained as 1, otherwise it is discarded and recorded as 0; the pixel points less than 1 / 2 of the adaptive threshold are directly recorded as 0, and the obtained binary image is the edge image.

[0119] In any of the above embodiments, further, in order to effectively remove linear and nonlinear noise of the image, a hybrid filter combining Gaussian filtering and adaptive median filtering is used to filter the grayscale image to generate a filtered image, specifically including:

[0120] Based on the preset filtering window, calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point;

[0121] Specifically, the size of the preset filter window is first set to a small size, such as 5×5, 10×10, and then the maximum size of the preset filter window is set, such as 20×20, 30×30. Then, the sliding window method is used to calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point in each sliding window when the sliding window slides, which are recorded as f min 、f max 、f med .

[0122] When the difference between the minimum grayscale value and the maximum grayscale value is determined to be less than the grayscale threshold, the grayscale value of the current center point is recorded as the center point grayscale value;

[0123] When it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, determine whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value. If so, record the grayscale median as the grayscale value of the center point. If not, increase the size of the preset filter window according to the preset step size, repeat the above process, and recalculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point;

[0124] Specifically, a grayscale threshold T is set, and based on the minimum grayscale value, maximum grayscale value, and grayscale median corresponding to the current center point in the window, the grayscale value of the center point corresponding to the window is determined. The process can be:

[0125] If f max -f min <T indicates that the gray value distribution of the pixel points in the window is uniform and there are no noise points, then the gray value f of the current center point xy is used as the gray value of the center point of the window;

[0126] If f max -f min ≥T and f min <f med <f max <, it indicates that there are noise points in the window, and the gray median value f med is not a noise point, then the gray median value f med is recorded as the gray value of the center point of this window;

[0127] If f max -f min ≥T, but does not satisfy f min <f med <f max <, then increase the size of the preset filtering window according to the preset step size, and repeat the above process.

[0128] It should be noted that in order to avoid the preset filtering window being too large, this embodiment also sets the maximum size of the preset filtering window. If the center point gray value still cannot be obtained when the preset filtering window is increased to the maximum size, then the average gray value of all points in the preset filtering window of the current maximum size is used as the gray value of the center point of this window.

[0129] Based on the center point gray value determined by the above process, a filtered image is generated.

[0130] In any of the above embodiments, further, the edge image includes a left edge image and a right edge image. In step 42, it specifically includes:

[0131] Convert the corrected second left image and the corrected second right image into grayscale images, and record them as the left grayscale image and the right grayscale image respectively;

[0132] Based on the left grayscale image and the right grayscale image respectively, in a sliding window manner, compare the gray value of each pixel in the window with the gray value of the center pixel of the window to generate a left gray 01 string and a right gray 01 string corresponding to the center pixel of the window;

[0133] Specifically, taking the left grayscale image as an example for illustration. In a sliding window manner, within a window centered on a certain pixel point, compare the gray value of each pixel in the window with the gray value of the center pixel, and mark 0 or 1 respectively according to the numerical size relationship, and connect them bit by bit to obtain a 01 string, that is, the left gray 01 string.

[0134] Based on the left edge image and the right edge image, respectively, a sliding window method is used to generate a left edge 01 string and a right edge 01 string of the center pixel of the window according to the distribution of the pixel points in the window as edge points;

[0135] Specifically, the sliding window method is also used. According to the binary left edge image, whether the pixel point in the window is an edge point - the edge point in the left edge image is recorded as 1, otherwise it is recorded as 0, and the bit-by-bit connection is obtained to obtain a 01 string, that is, the left edge 01 string.

[0136] The left grayscale 01 string, the right grayscale 01 string, the left edge 01 string, and the right edge 01 string are respectively concatenated and used as the code of the central pixel to generate the left code and the right code of the central pixel of the window;

[0137] Based on the Hamming distance between the left code and the right code, that is, the number of bits with different digits in the two code strings, the Census value is calculated, and the AD-Census cost value is constructed based on the Census value, the corrected second left image and the corrected second right image.

[0138] Embodiment 2:

[0139] like Figure 2 As shown, this embodiment provides a binocular camera parallax calculation system 100 based on illumination reference, and the system 100 includes:

[0140] A first disparity calculation unit 101, the first disparity calculation unit 101 is configured to: obtain a first left image and a first right image captured by a binocular camera under a first illumination condition, and calculate a corresponding first disparity map;

[0141] The feature calculation unit 102 is configured to: mark the static target area in the first left image based on the first disparity map, and extract the illumination reference feature of the image corresponding to the static target area in the first left image;

[0142] The correction unit 103 is configured to: perform illumination correction on the second left image and the second right image captured by the binocular camera under the second illumination condition based on the illumination reference feature;

[0143] The second disparity calculation unit 104 is configured to determine a second disparity map based on the corrected second left image and the corrected second right image, wherein the second lighting condition is a weak lighting condition and / or a strong lighting condition.

[0144] In any of the above embodiments, further, the feature calculation unit 102 is also configured to:

[0145] Clustering the first disparity map to determine a clustering area;

[0146] When it is determined that a change in pixel coordinates of any cluster region in the first disparity map of multiple consecutive frames is less than or equal to a static threshold, the cluster region is marked as a static disparity region;

[0147] Based on the pixel coordinates of the static parallax area, the first left image is marked, and the marked area is recorded as the static target area.

[0148] In any of the above embodiments, further, the correction unit 103 is also configured as follows: Step 31: Based on the static target area, extract the illumination features of the image at the corresponding position of the second left image to obtain the current illumination features; Step 32: Based on the illumination reference features and the current illumination features, calculate the regional incremental index corresponding to each static target area under the second illumination condition; Step 33: Calculate the mean of the regional incremental index, and use the mean to perform illumination correction on the second left image and the second right image.

[0149] In any of the above embodiments, further, the second disparity calculation unit 104 is further configured to:

[0150] determining an edge image based on the corrected second left image and the corrected second right image;

[0151] constructing an AD-Census cost value based on the edge image, the corrected second left image, and the corrected second right image;

[0152] Based on the disparity value corresponding to the static target area, the AD-Census cost value is optimized, and based on the optimized AD-Census cost value, the fixed path aggregation cost value is determined, where the calculation formula of the optimized AD-Census cost value is:

[0153]

[0154] Where C(p, d) is the optimized AD-Census cost value, C0(p, d) is the AD-Census cost value, p is the pixel point in the corrected second left image, d is the disparity value of the pixel point p in the corrected second right image, and d′ is the static target area StOR j The corresponding disparity value, λ is the optimization coefficient;

[0155] Based on the fixed path aggregation cost value, a global energy function is constructed, and the disparity value corresponding to the minimum value of the global energy function is recorded as the disparity value of the pixel point p;

[0156] The pixel points in the corrected second left image are traversed to determine a disparity map.

[0157] In any of the above embodiments, further, the edge image includes a left edge image and a right edge image, which are respectively determined by the corrected second left image and the corrected second right image, and the second disparity calculation unit 104 is also configured to: convert the corrected second left image into a grayscale image, and filter the grayscale image to generate a filtered image; calculate an adaptive threshold for each pixel in the filtered image, wherein the adaptive threshold is determined by the grayscale value of the pixel and the corresponding weight, and the weight is determined by the absolute value of the difference between the grayscale value of the pixel and the grayscale mean in the corresponding pixel window; based on the adaptive threshold, the Sobel edge detection algorithm is used to determine the left edge image.

[0158] In any of the above embodiments, further, the second disparity calculation unit 104 is also configured to: calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point based on the preset filtering window; when it is determined that the difference between the minimum grayscale value and the maximum grayscale value is less than the grayscale threshold, record the grayscale value of the current center point as the center point grayscale value; when it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, determine whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value, if so, record the grayscale median as the center point grayscale value, if not, increase the size of the preset filtering window according to the preset step size, and recalculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; generate a filtered image based on the center point grayscale value.

[0159] In any of the above embodiments, further, the edge image includes a left edge image and a right edge image, and the second disparity calculation unit 104 is further configured to: convert the corrected second left image and the corrected second right image into grayscale images, which are respectively recorded as a left grayscale image and a right grayscale image; based on the left grayscale image and the right grayscale image, respectively, using a sliding window method, the grayscale value of each pixel in the window is compared with the grayscale value of the pixel at the center of the window, and a left grayscale 01 string and a right grayscale 01 string corresponding to the pixel at the center of the window are generated; based on the left edge image and the right edge image, respectively, For an image, a sliding window method is used to generate a left edge 01 string and a right edge 01 string of the center pixel of the window according to the distribution of the pixels in the window as edge points; the left grayscale 01 string and the right grayscale 01 string are respectively concatenated with the left edge 01 string and the right edge 01 string to generate the left code and the right code of the center pixel of the window; the Census value is calculated based on the Hamming distance between the left code and the right code, and the AD-Census cost value is constructed based on the Census value, the corrected second left image and the corrected second right image.

[0160] In any of the above embodiments, further, the calculation formula of the AD-Census cost value is:

[0161] C0(p,d)=ρ(AD,λAD )+ρ(Census,λ census )

[0162]

[0163] In the formula, C0(p,d) is the AD-Census cost, ρ() is the mapping function, AD is the difference in the absolute value of the pixel, Census is the Census value, which indicates the similarity of the relative relationship between the adjacent areas of the pixel, and λ AD , census is the preset parameter, is the value of the q channel in the RGB three channels corresponding to the pixel point p in the corrected second left image, is the value of the q channel in the RGB three channels corresponding to the pixel point after the corresponding pixel point p in the corrected second right image is offset by the parallax d.

[0164] Embodiment 3:

[0165] This embodiment also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executes the execution instructions stored in the memory, so that the processor or other hardware module executes the above method.

[0166] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0167] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0168] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0169] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A binocular camera parallax calculation method based on illumination reference, characterized in that: The method comprises: Step 1: Obtain a first left image and a first right image captured by a binocular camera under a first illumination condition, and calculate a corresponding first disparity map; Step 2: Based on the first disparity map, annotate the static target area in the first left image, and extract the illumination reference features of the corresponding area image of the static target area in the first left image; Step 3: Based on the illumination reference feature, performing illumination correction on the second left image and the second right image captured by the binocular camera under the second illumination condition; Step 4: Determine a second disparity map based on the corrected second left image and the corrected second right image, wherein the second lighting condition is a weak lighting condition and / or a strong lighting condition.

2. The binocular camera parallax calculation method based on illumination reference according to claim 1, characterized in that: The marking of the static target area in the first left image based on the first disparity map specifically includes: Step 21: clustering the first disparity map to determine a clustering area; Step 22: when it is determined that the change in the pixel coordinates of any cluster area in the first disparity map of multiple consecutive frames is less than or equal to a static threshold, marking the cluster area as a static disparity area; Step 23: Based on the pixel coordinates of the static parallax area, mark the first left image, and record the marked area as the static target area.

3. The binocular camera parallax calculation method based on illumination reference according to claim 1 or 2, characterized in that: The step 3 specifically includes: Step 31: Based on the static target area, extract the illumination feature of the image at the corresponding position of the second left image to obtain the current illumination feature; Step 32: Calculating the regional increment index corresponding to each of the static target areas under the second illumination condition based on the illumination reference feature and the current illumination feature; Step 33: Calculate the mean of the regional increment index, and use the mean to perform illumination correction on the second left image and the second right image.

4. The binocular camera parallax calculation method based on illumination reference according to claim 1, characterized in that: The step 4 specifically includes: Step 41: determining an edge image based on the corrected second left image and the corrected second right image; Step 42: constructing an AD-Census cost value based on the edge image, the corrected second left image and the corrected second right image; Step 43: Based on the disparity value corresponding to the static target area, the AD-Census cost value is optimized, and based on the optimized AD-Census cost value, a fixed path aggregation cost value is determined, wherein the calculation formula of the optimized AD-Census cost value is: Wherein, C(p, d) is the optimized AD-Census cost value, C0(p, d) is the AD-Census cost value, p is the pixel point in the corrected second left image, d represents the disparity value of the pixel point p offset in the corrected second right image, and d ′ is the static target area StOR j The corresponding disparity value, λ is the optimization coefficient; Step 44: constructing a global energy function based on the fixed path aggregation cost value, and recording the disparity value corresponding to the minimum value of the global energy function as the disparity value of the pixel point p; Step 45: traverse the pixel points in the corrected second left image to determine the disparity map.

5. The binocular camera parallax calculation method based on illumination reference as claimed in claim 4, characterized in that: The edge image includes a left edge image and a right edge image, which are respectively determined by the corrected second left image and the corrected second right image. In step 41, the method for determining the left edge image specifically includes: Converting the corrected second left image into a grayscale image, and filtering the grayscale image to generate a filtered image; Calculating an adaptive threshold for each pixel in the filtered image, wherein the adaptive threshold is determined by the grayscale value of the pixel and a corresponding weight, and the weight is determined by the absolute value of the difference between the grayscale value of the pixel and the grayscale mean in the corresponding pixel window; Based on the adaptive threshold, the Sobel edge detection algorithm is used to determine the left edge image.

6. The binocular camera parallax calculation method based on illumination reference as claimed in claim 5, characterized in that: The filtering of the grayscale image to generate a filtered image specifically includes: Based on the preset filtering window, calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; When it is determined that the difference between the minimum grayscale value and the maximum grayscale value is less than the grayscale threshold, the grayscale value of the current center point is recorded as the center point grayscale value; When it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, determine whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value. If so, record the grayscale median as the grayscale value of the center point. If not, increase the size of the preset filter window according to the preset step size, and recalculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point. The filtered image is generated based on the grayscale value of the center point.

7. The binocular camera parallax calculation method based on illumination reference as claimed in claim 4, characterized in that: The edge image includes a left edge image and a right edge image. The step 42 specifically includes: Converting the corrected second left image and the corrected second right image into grayscale images, which are respectively referred to as a left grayscale image and a right grayscale image; Based on the left grayscale image and the right grayscale image, respectively, a sliding window method is used to compare the grayscale value of each pixel in the window with the grayscale value of the central pixel of the window, and a left grayscale 01 character string and a right grayscale 01 character string corresponding to the central pixel of the window are generated; Based on the left edge image and the right edge image respectively, a sliding window method is used to generate a left edge 01 string and a right edge 01 string of the center pixel of the window according to the distribution of the pixel points in the window as edge points; Respectively concatenate the left grayscale 01 string, the right grayscale 01 string, the left edge 01 string, and the right edge 01 string to generate a left code and a right code of the center pixel of the window; A Census value is calculated based on the Hamming distance between the left code and the right code, and the AD-Census cost value is constructed based on the Census value, the corrected second left image, and the corrected second right image.

8. The binocular camera parallax calculation method based on illumination reference according to claim 4 or 7, characterized in that: The calculation formula of the AD-Census cost value is: C0(p,d)=ρ(AD,λ AD )+ρ(Census,λ census ) Where C0(p,d) is the AD-Census cost, ρ() is the mapping function, AD is the difference in absolute values ​​of pixels, Census is the Census value, which indicates the similarity of the relative relationship between adjacent regions of pixels, and λ AD , census is the preset parameter, is the value of the q channel in the RGB three channels corresponding to the pixel point p in the corrected second left image, is the value of the q channel in the RGB three channels corresponding to the pixel point after the corresponding pixel point p in the corrected second right image is offset by the parallax d.

9. A binocular camera parallax calculation system based on illumination reference, characterized in that: The system comprises: A first disparity calculation unit, wherein the first disparity calculation unit is configured to: obtain a first left image and a first right image captured by a binocular camera under a first illumination condition, and calculate a corresponding first disparity map; a feature calculation unit, the feature calculation unit being configured to: mark a static target area in the first left image based on the first disparity map, and extract a lighting reference feature of an area image corresponding to the static target area in the first left image; A correction unit, wherein the correction unit is configured to: perform illumination correction on a second left image and a second right image captured by the binocular camera under a second illumination condition based on the illumination reference feature; A second disparity calculation unit, wherein the second disparity calculation unit is configured to determine a second disparity map based on the corrected second left image and the corrected second right image, wherein the second lighting condition is a weak lighting condition and / or a strong lighting condition.

10. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the binocular camera parallax calculation method based on illumination reference according to any one of claims 1 to 7.

Images