[0057] The general flow chart of the front vehicle detection method based on monocular vision of the present invention is as follows figure 1 As shown, firstly, the grayscale image of the road conditions in front of the vehicle is preprocessed, and the bottom shadow of the vehicle is segmented from it, and then fake vehicles are filtered out by the position and geometric characteristics of the shadow at the bottom of the vehicle, and finally the unstable target is filtered out through the target tracking decision. The final vehicle inspection result. The implementation process of the present invention will be further detailed below in conjunction with the accompanying drawings.
[0058] 1. Image acquisition and preprocessing
[0059] (1) Original image acquisition
[0060] Use CMOS black and white industrial camera to collect grayscale images in real time. Set the industrial camera parameters so that the frame rate of the captured video reaches 25frames/s, and the resolution of the collected grayscale image is 640×480. Reference figure 2 After installing the industrial camera on the front windshield of the car, it is located in the center of the center console so that d1=d2, facing the front so that the angle α=0 degrees, the horizontal height h is about 1.2m, and the depression angle β is about 15 degrees. image 3 It is a frame of original image collected.
[0061] (2) Image preprocessing
[0062] by image 3 It can be seen that the original image collected has a high resolution and the pixels containing vehicle information are all concentrated in the lower part of the image. Directly processing the entire frame will increase the amount of calculation and affect the real-time performance, and the grayscale range of the image is small. It is easy to process images.
[0063] To solve the above problems, do the following processing on the original image:
[0064] ① Intercept the area containing the bottom information of the vehicle in the original image I, that is, intercept the lower 1/3 area of the original image I that contains the borders on both sides to obtain the sub-image Ipart, and copy the sub-image Ipart to obtain the image Ipart1;
[0065] ②Using the histogram equalization method to stretch the gray value of the sub-image Ipart to 256 gray levels, the result is as follows Figure 4 Shown.
[0066] 2. Image segmentation
[0067] The preprocessed image is divided into background and target (shadow on the bottom of the vehicle). Since the pixel value of the shadow at the bottom of the vehicle is smaller than the pixel value of the surrounding pixels, an appropriate threshold can be used to separate the shadow at the bottom of the vehicle from the background. Pixels below the threshold are determined as the bottom shadow of the vehicle, and pixels above the threshold are determined as the background. This patent uses the following method to obtain a threshold to segment the image after histogram equalization.
[0068] (1) Acquisition of segmentation threshold
[0069] First, extract the average gray value of the road surface in a small rectangular area Iroad intercepted in the sub-image Ipart. The vertical center line of the rectangular area Iroad coincides with the vertical center line of the sub-image Ipart, and the bottom boundary line of the rectangular area Iroad coincides with the bottom boundary of Ipart. Combined with the method of fixing the industrial camera in this patent, the experimental conclusions obtained from a large number of images collected prove that there are generally no obstacles such as vehicles and pedestrians in a certain area in front of the vehicle, that is, the area Iroad. Therefore, only need to remove With the influence of lane lines with higher gray values, the gray level of the road can be extracted from the area.
[0070] The steps to calculate the average gray value of the road are as follows:
[0071] ①Calculate the average gray value Ta of all points in Iroad;
[0072] ② Divide all points in Iroad into two categories [0, Ta] and [Ta, 255];
[0073] ③Calculate the average gray value of points in the range of [0, Ta] as the average gray value Tr of the road gray.
[0074] Then, the improved OTSU method is used to extract the gray threshold for the sub-image Ipart, that is, the traditional Otsu method (OTSU) is used to extract the temporary gray-level threshold t for the sub-image Ipart after the gray-level histogram equalization, so that the entire gray-scale range Divided into two parts [0, t] and [t, 255], and then within the gray range of [t, 255], use the Otsu method again to obtain the gray threshold To; the specific method is as follows:
[0075] Suppose the number of pixels of a grayscale image is N, there are L gray levels, and the number of pixels with gray level i is n i , Then The histogram is expressed as a probability density distribution.
[0076] P i = n i N , X i = 0 L - 1 P i = 1 , P i ≥0
[0077] Let t be the segmentation threshold between the foreground and the background, and the proportion of the number of front spots in the image is ω 0 , The average gray scale is μ 0;The proportion of background points in the image is ω 1 , The average gray scale is μ 1. Use the threshold t to divide the gray levels into two categories: C 0 =(0,1,...,t) and C 1 = (T+1, t+2,..., L-1). C 0 And C 1 The occurrence probability and mean value of are respectively:
[0078] ω 0 = X i = 0 t P i = ω ( t )
[0079] ω 1 = X i = t + 1 256 - t P i = 1 - ω ( t )
[0080] μ 0 = X i = 0 t iP i / ω 0 = μ ( t ) / ω ( t )
[0081] μ 1 = X i = t + 1 256 - t iP i / ω 1 = μ T ( t ) - μ ( t ) 1 - ω ( t )
[0082] among them
[0083] ω ( t ) = X i = 0 t iP i
[0084] μ T ( t ) = X i = 0 255 iP i
[0085] The variance between classes is:
[0086] σ B 2 = ω 0 ( μ 0 - μ T ) 2 + ω 1 ( μ 1 - μ T ) 2 = ω 0 ω 1 ( μ 1 - μ 0 ) 2
[0087] Then the threshold t can be obtained by Is the maximum value of
[0088] σ B 2 ( thre ) = max 0 ≤ t ≤ 255 { σ B 2 ( t ) }
[0089] The above method is used for the sub-image Ipart, that is, the OTSU method extracts the gray threshold to divide the entire gray range into two parts [0, t] and [t, 255], and then in the gray range of [0, t], Use the OTSU method again to get the gray threshold To.
[0090] Finally, the dual-threshold linear binarization method is used to obtain the current frame binarization threshold. The method is as follows:
[0091] Tr and To are the average gray value of the road surface and the threshold value obtained using the OTSU method, respectively. Let H be the resolution in the vertical direction of the image, and f(i,j) is the gray value of the pixel in the i-th row and j-th column, Th i Is the binarization threshold of the i-th row.
[0092] Th i =α×To+β×Tr
[0093] Among them, α=i/H, β=(H-i)/H;
[0094] If f(i,j) i , It is divided into the target; otherwise, it is divided into the background.
[0095] (2) Evaluation of binarization results
[0096] According to the characteristics of spatial vision, targets farther from the viewpoint occupy a smaller pixel area in the image, and targets closer to the viewpoint occupy a larger pixel area in the image, so they are generally segmented farther away The target occupies a small number of pixels, while the closer target occupies a relatively large number of pixels. Therefore, the sub-image Ipart is first divided into two regions, Ifar and Inear. Ifar is the one corresponding to the far road. Image area, that is, the upper half of the area including the borders on both sides of the sub-image Ipart, Inear is the image area corresponding to the closer road, that is, the lower half of the area including the borders on both sides of the sub-image Ipart;
[0097] Let p(i,j) be the gray value of the pixel in the i-th row and j-th column, Th i Is the binarization threshold of the i-th row, the upper limit of the binarization evaluation of the corresponding area Ifar is EvaluFarUp, the lower limit is EvaluFarLow, and the binarization evaluation limit of the image area Inear corresponding to the closer road is EvaluNear, above Binarization evaluation limits are all empirical constants;
[0098] ① Find the regions Ifar and Inear satisfy p(i,j) i The number of pixels NumFar and NumNear;
[0099] ②If NumFar>EvaluFarUp, it means that the current frame threshold is too large and the segmentation result is too noisy. At this time, the preliminary threshold is used to segment the current frame;
[0100] ③If NumFar EvaluNear, it means that there is too much noise in the segmentation result in the area Inear, and the current frame is segmented by the preliminary threshold value, and the target in the area Ifar is enhanced;
[0101] ④ If NumFar
[0102] ⑤If NumFar
[0103] The adjustment value δt is an empirical constant, and the target in the area Ifar is enhanced.
[0104] The method for obtaining the preliminary threshold is as follows:
[0105] ⑤ If the previous frame threshold Tp is less than the current frame threshold Tc, the previous frame threshold Tp is used as the preliminary threshold;
[0106] ⑥ If the previous frame threshold Tp is greater than or equal to the current frame threshold Tc, reduce the current frame segmentation threshold to Tc-δt as the preliminary threshold;
[0107] The preliminary thresholds of Tr and To are both obtained by the above method.
[0108] (3) Image segmentation and enhancement processing
[0109] The binarization result is used to evaluate the judgment thresholds Tr and To, and the double-threshold linear binarization method is used to segment the image after histogram equalization. The segmentation threshold Th corresponding to the pixel in the i-th row i , f ( i , j ) = 0 f ( i , j ) ≥ Th i 255 f ( i , j ) Th i , f(i, j) is the gray value of the pixel in the i-th row and j-th column in the sub-image Ipart. Then, according to the judgment result, the target in the area Ifar is enhanced, that is, the target is enhanced in the vertical and horizontal directions. The processing steps are as follows:
[0110] ① Traverse the target pixel point p(i,j) down the j-th column to point p′(i+h,j). If the target is encountered during the traversal process, it means that the target pixel is set to p”(i +h′,j), where 1≤h′≤h, then the pixel values of the points between the pixel points p(i,f) and p″(i+h′j) are all set to 255;
[0111] ②The principle is the same as above. Traverse the target pixel point p(i,j) to the left along the i-th row to point p′(i,jw). If the target is encountered during the traversal process, it means that the target pixel is set to p″( i, jw′), where 1≤w′≤w, then the pixel values of the points between the pixel points p(i,j) and p″(i,jw′) are all set to 255, where w=a×i +b, a and b are empirical constants.
[0112] Figure 5 Ibinary is a binary result graph.
[0113] 3. Get current frame target information
[0114] (1) Noise filtering
[0115] The area of the vehicle on the road often has a large gray gradient vector modulus. This feature can be used to extract the area of the image where the vehicle may exist. The specific method is as follows:
[0116] Move the window W of St×St on the image Ipart1 with St as the step size, and calculate the gray gradient vector modulus of the image in the window; the gradient modulus is | ▿ f ( x , y ) | = [ ( ∂ f / ∂ x ) 2 + ( ∂ f / ∂ y ) 2 ] 1 2
[0117] Substituting difference for differentiation, the approximate expression of gradient modulus is
[0118] | ▿ f ( x , y ) | = { [ f ( x , y ) - f ( x + 1 , y ) ] 2 + [ f ( x , y ) - f ( x , y + 1 ) ] 2 } 1 2
[0119] Calculate the sum Gw of the gray gradient vector modulus of the pixel in the window W
[0120] Gw = X ( x , y ) A w { [ f w ( x , y ) - f w ( x + 1 , y ) ] 2 + [ f w ( x , y ) - f w ( x , y + 1 ) ] 2 } 1 2
[0121] Then the gray value of all pixels in window W at this time
[0122] f ( x , y ) = 255 | ▿ f ( x , y ) | ≥ G 0 | ▿ f ( x , y ) | G
[0123] The modulus detection threshold G is an empirical constant. Image 6 Is the image Ipart1 processed by this method.
[0124] The image Ipart1 processed by this method and the binary image Ibinary are ANDed to obtain the binary image Icalibration, such as Figure 7 As shown, the method is as follows:
[0125] 1) Two points f with the same horizontal and vertical coordinates on the two images 1 (i, j), f 2 (i, j);
[0126]
[0127] Using spatial vision characteristics, combined with the fixing method and parameter setting of the industrial camera in this embodiment, filtering is performed in a progressive scanning manner. Let the filter range of the i-th row be R i =[Wlow, WHigh], where WLow=a×i, WHigh=b×i+c, a, b, and c are all empirical constants. Progressive scanning retains the target pixels that meet the filtering conditions. For those that do not meet the filtering conditions, The target pixel of is filtered out, that is, the pixel value of this point is set to 0. Figure 8 for Figure 7 The effect picture obtained after noise filtering.
[0128] (2) Get the position line
[0129] Scan column by column from bottom to top. When the target pixel point p(i,j) is encountered, continue to scan upward from point p(i-1,j), if the next target pixel point p(id,j) is detected When the distance between the two targets is greater than δd (δd is a constant summed up by combining the experimental results of the fixing method and parameter setting of the industrial camera in this embodiment), then the points p(i,j) and p(id,j) All are reserved as position lines, and the pixel values of other points in the current column are all set to 0. Picture 9 for Figure 8 The position line result graph.
[0130] (3) Line fusion
[0131] Scan line by line from right to left, when traversing to the left end point p(i,j) of the position line, and the position line satisfies a certain length δL, that is, the distance from point p(i,j) to point p(i,j-δL) When the pixel value is 255, c=(1,2,3,...), then set the pixel values of point p(i-1, j-δL-c) and point p(i+1, j-δL-c) to 0, The fusion processing of this position line until the pixel values of points p(i,j-δL-c), p(i-1,j-δL-c), p(i+1,j-δL-c) Both are up to 0. In this way, the purpose of fusing the position lines of the same target in different pixel rows into one line is achieved. Where δL is an empirical constant. Picture 10 for Picture 9 The line fusion result graph. Perform noise filtering again on the sub-image Ipart after the line fusion processing, and get the result Picture 11.
[0132] (4) Obtain target information
[0133] According to the safe distance between vehicles stipulated in the Road Traffic Safety Law of the People’s Republic of China and the experimental summary of the fixing method and parameter setting of the industrial camera in this embodiment, it can be obtained that the vehicle that is on the road ahead poses a threat to the driving safety of the vehicle. The position lines on the video image are all concentrated in area D. Such as Picture 12 As shown, the area D is a trapezoidal area of the upper and lower borders of Ipart, which is intercepted on the sub-image Iprat. Therefore, only the length, position and other information of the position line where the midpoint of the position line is located in the area D is left as the current frame target detection result CurResult for subsequent processing. Figure 13 Is the result after filtering in area D, Figure 14 It is the result display of CurResult on the original image.
[0134] 4. Target tracking judgment
[0135] Target tracking judgment, the purpose of the present invention’s monocular vision-based forward vehicle detection method is to filter out the single-frame false targets that suddenly appear, and to compensate for the missed detection during the tracking period for targets that have been tracked stably for a long period of time, Thereby improving the robustness of the vehicle detection system ahead.
[0136] (1) Target information matching
[0137] The matching method between targets is as follows:
[0138] Let L be the length of the target position line, Pt_le is the left end point of the target position line, Pt_ri is the right end point of the target position line, and i is the line of the candidate target position line of the current frame,
[0139] δX=|Pt_le.x-Pt_ri.x|
[0140] δY=|Pt_le.y-Pt_ri.y|
[0141] δL=|L 1 -L 2 |
[0142] The limit of δX is:
[0143] mX=a×i+b
[0144] The limit of δY is:
[0145] mY=c×i+d
[0146] The limit of δL is:
[0147] mL=e
[0148] Among them, a, b, c, d, and e are all empirical constants.
[0149] When δX≤mX, δY≤mY, and δL≤mL at the same time, the target position and size match successfully. Then calculate the similarity between the target and the candidate target histogram, and the target whose similarity meets the requirements is the target result CurResult of the current frame.
[0150] (2) Target classification judgment
[0151] The target result CurResult of the current frame is matched with the target result PrevResult of the previous frame and the tracking target result TrackResult, and the target result of the current frame is classified and judged according to the matching result. The steps are as follows:
[0152] ① CurResult fails to match with PrevResult and TrackResult, then the CurResult is a new target that appears for the first time, and its information is retained and sent to the PrevResult collection, but this frame is not displayed as the vehicle detection result output. If it fails to match the next frame, it is judged as Single frame false target FalseResult is removed;
[0153] ② CurResult and PrevResult match successfully, but CurResult and TrackResult fail to match, then the CurResult is judged as the newly appeared target NewResult, and its information is output and displayed as the vehicle detection result, and then NewResult is added to the TrackResult collection;
[0154] ③CurResult matches PrevResult and TrackResult successfully, then the CurResult is judged to be a stable tracking target TrackResult, and its information is output and displayed as the vehicle detection result, and the TrackResult information is updated. Its reliability QR increases by δq, and δq is a constant;
[0155] ④CurResult and PrevResult fail to match, but CurResult and TrackResult match successfully, then the CurResult is judged to be the tracked target TrackResult, its information is output and displayed as the vehicle detection result, and the TrackResult information is updated;
[0156] ⑤For the target that fails to match CurResult in the TrackResult set, but the reliability QR ≥ α, and the closest distance between the midpoint of the position line and the boundary line of the area D is not less than δr, it is judged as the second-class tracking target result InfResult, and its information is The vehicle detection result is output and displayed, and then its reliability QR is reduced by δq. If there is still no matching success in subsequent t frames, or the closest distance between the midpoint of the position line and the boundary line of area D is less than δr, then the target information will be taken from TrackResult Excluded from the set, where δr, α, t are empirical constants.
[0157] Such as Figure 15 , Figure 16 For the debug image after the target tracking decision processing, the upper 1/3 of the debug image with black as the background image is the Ipart vehicle position line binary result image overlaid on the gray result image. Figure 15 The target C in the middle box is judged to be a false target FalseResult after tracking and classification, Figure 16 The middle target A is the secondary tracking target result InfResult, the target D is the newly appeared target NewResult, and the target B is the stable tracking target TrackResult.
[0158] Figure 15 , Figure 16 The target marked with a line in the mid-gray image part is the final detection result of the front vehicle detection method based on monocular vision of the present invention.
[0159] The front vehicle detection method adopted by the present invention makes full use of the shadow characteristics of the bottom of the vehicle, the gray gradient characteristics of the vehicle, and the position characteristics of the target vehicle continuously appearing in the video sequence, etc., thereby realizing that it can accurately detect the front of the vehicle on the road. Detection methods for vehicles that pose a safety threat.
