An automatic detection method for highway height restriction facilities based on binocular vision

Abstract

The present invention belongs to the field of active safety of automobile automatic driving, and provides a method for automatic detection of highway height restriction facilities based on binocular vision. Aiming at the problems of false detection and low accuracy of long-distance height measurement caused by the diversity of highway height restriction scenes and the variability of driving conditions, the accuracy of height restriction target detection is improved by optimizing the distribution of each sample of a binocular height restriction data set; local operators and global operators are used at the output end to improve the estimation accuracy of the distance and height of the height restriction target, and the distance and height of the height restriction facility are output in real time. The method provides assistance to large vehicle drivers when they pass through the height restriction section through a real-time visual interface display, and effectively reduces unnecessary vehicle damage and public property loss caused by misjudgment of drivers due to lack of experience when passing through highway height restriction facilities.

Description

Technical Field

[0001] The present invention belongs to the field of active safety of automobile automatic driving, and in particular relates to a method for real-time height and distance measurement of highway height limit facilities based on binocular vision sensors. Background Art

[0002] In recent years, with the further improvement of the domestic economic level, RV self-driving has gradually become known to more and more people with the explosive growth of short videos, becoming a popular way of travel. In addition to RVs, large trucks, large buses and special vehicles can be seen everywhere in various scenes of our daily lives. In order to ensure the normal operation of traffic in the city, various cities across the country have set up a large number of height-limiting facilities at the entrances to urban areas and highway entrances to prevent large vehicles from entering the city and hindering traffic. However, since some height-limiting facilities in China do not have unified national standards and specifications, the height-limiting facilities set up by local governments vary greatly in appearance, height limit and other aspects, which has brought many troubles to the normal passage of large vehicles.

[0003] In the past few years, height limit accidents involving large vehicles have occurred frequently due to the irregular setting of height limit facilities and driver fatigue driving. For height limit facilities such as bridges and tunnels, which are very expensive to build, once an accident occurs, the losses to public property are huge; for large buses and large trucks, the occurrence of height limit accidents will also cause huge losses to personal property and life safety. Therefore, a solution that can output the distance to the height limit facility in front of the road and the height of the height limit facility in real time has very important research significance and practical value for the active safety of large vehicles.

[0004] At present, the mainstream solutions for high-range measurement in the market mainly include millimeter-wave radar, laser radar and binocular camera. Among them, the advantage of millimeter-wave radar ranging is that it has the functions of ranging and speed measurement, has a long effective detection distance, and is cost-effective in the field of vehicle-mounted ranging, but its disadvantage is that it has weak angular resolution and cannot distinguish the details of objects. The detection sensitivity of metals is higher than that of non-metallic materials, and the detection effect of people is not good in scenes where people and cars are mixed; the advantage of laser radar is long distance measurement, excellent angular resolution, little influence by light and no need for deep learning algorithms, but its disadvantage is that it is very expensive and greatly affected by bad weather; the advantage of binocular camera is low cost, mature hardware research and development technology and rich information contained in image data, which is conducive to target recognition, but its disadvantage is that it relies heavily on parallax and is greatly affected by backlight and bad weather.

[0005] Among visual sensors, the cost of monocular cameras is lower than that of binocular cameras. However, due to the scale uncertainty of monocular ranging, only the relative depth of the target to be measured can be obtained, so the ranging accuracy is much lower than that of binocular cameras. RGB-D depth cameras measure the depth of the target to be measured through structured light or TOF (Time of flight) physical methods, but this ranging method is limited by the characteristics of narrow measurement range, high noise, small field of view and susceptibility to sunlight interference, and is mainly used indoors. Therefore, the present invention proposes a real-time monitoring solution for highway height limit equipment based on binocular sensors, so that RVs, large trucks and other special vehicles can achieve real-time height measurement and ranging within a certain distance range before passing through a series of highway height limit facilities such as height limit poles, bridge holes and tunnels, so as to achieve the effect of early warning.

[0006] Since disparity calculation consumes a lot of computing resources, based on real usage scenarios and lightweight design ideas, this method adopts a two-dimensional target detection method to replace the entire image with a local detection area as the input of disparity calculation. In the height measurement and ranging output judgment, a combination of local operators and global operators is adopted to reduce the amount of algorithm calculation and improve the real-time performance of the algorithm while ensuring the accuracy of height measurement and ranging. Through the early warning system, it effectively reduces unnecessary vehicle damage and property losses caused by misjudgment due to fatigue driving or lack of experience when drivers of large vehicles pass through highway height limit facilities. Summary of the invention

[0007] The present invention provides a method for full-automatic height limit detection and height and distance measurement based on binocular vision. Aiming at the problems of false detection, missed detection and low accuracy of long-distance height measurement caused by the diversity of highway height limit scenes and the variability of driving conditions, the accuracy of height limit target detection is improved by adding negative sample data in a training set, optimizing the distribution of samples of each scene in a binocular height limit data set, and other measures; in the parallax calculation stage, a plurality of filtering methods are used to improve the credibility of parallax data, thereby improving the accuracy of height limit target parallax calculation; local operators and global operators are used at the output end to improve the estimation accuracy of the distance and height of the height limit target, and the distance of the height limit target from the driving vehicle and the height of the height limit target are output in real time, so as to assist the driver in driving safety.

[0008] The idea of ​​this method is: collect and annotate height limit data to construct a binocular height limit data set; perform data enhancement on some height limit scenes so that the distribution of training samples of each height limit scene in the training set is relatively balanced; add negative samples to the training set to reduce the false detection rate of target detection; adjust the distribution of samples of each scene in the data set to reduce the missed detection rate of target detection; use the two-dimensional detection frame output by target detection to crop the disparity data as the input of the disparity filtering stage; perform coarse-grained filtering and fine-grained filtering operations on the cropped disparity data to obtain the disparity calculation value of the height limit target and the bottom edge position information of the height limit target; calculate the distance between the height limit target and the camera and the height of the height limit target; use local and global control parameters to judge the rationality of the calculated distance and height information to ensure the rationality and robustness of the distance and height output. This method can meet the accuracy requirements of distance and height measurement of height limit targets within a safe braking distance.

[0009] The technical solution of the present invention is:

[0010] A method for automatically detecting highway height restriction facilities based on binocular vision comprises the following steps:

[0011] Step 1: Collect binocular height-limited scene data, complete data annotation, and construct a binocular height-limited dataset;

[0012] (1.1) Collect diverse highway binocular height restriction scene data, including height restriction facilities such as height restriction poles, bridge holes, archways, and tunnels;

[0013] (1.2) Classify highway binocular height restriction scene data according to different height restriction facilities and different appearance features of similar height restriction facilities, add negative sample data with appearance features similar to those of height restriction facilities to the classified data, and obtain highway binocular height restriction scene data;

[0014] (1.3) Using annotation software to annotate the highway binocular height restriction scene data in step (1.2); obtaining a binocular height restriction data set;

[0015] (1.4) Using the application programming interface provided by the binocular camera, obtain the disparity data of the left camera and the right camera after stereo matching in the highway binocular height limit scene data;

[0016] Step 2, using the highway binocular height limit scene data and annotation information in the binocular height limit dataset to train the target detection model and obtain a height limit target detector;

[0017] (2.1) Perform data enhancement on the highway binocular height limit dataset so that the number of samples of each height limit scene in the binocular height limit dataset is evenly distributed;

[0018] (2.2) Input the left camera image and the annotation information corresponding to the left camera image of the data-enhanced binocular height-limited dataset into the selected object detection network for training to obtain a height-limited object detector;

[0019] Step 3: Take the left eye sequence image data of the binocular camera as input, combine it with the disparity data corresponding to the binocular image pair in the binocular height limit data set, and output the disparity information sequence of the detection area of ​​the height limit target as the input of the next stage;

[0020] (3.1) The left-eye sequence image data of the binocular camera is used as input, and is first processed by the height-limited target detector obtained in step (2.2), and the left-eye sequence image data and the two-dimensional detection frame of the height-limited target area are output, including the position information of the two-dimensional detection frame and the category information of the height-limited facility;

[0021] (3.2) using the position information of the two-dimensional detection frame output by the detection in step (3.1) to crop the disparity data obtained in step (1.4) to obtain the disparity sequence data of the detection area;

[0022] Step 4, performing coarse-grained filtering and fine-grained filtering on the disparity sequence data of the detection area obtained in step 3, calculating the disparity value of the height-limited target and locating the pixel coordinate information corresponding to the bottom edge of the height-limited target;

[0023] (4.1) Since the parallax size is inversely proportional to the distance between the height-limited target and the camera optical center, the outlier data threshold lower bound Alpha and the outlier data threshold upper bound Beta are set to perform rough cleaning on the parallax sequence data output from step (3.2);

[0024]

[0025] Where Z is the detection distance between the height-limited facility and the camera; baseline is the baseline of the binocular camera; f is the focal length of the binocular camera; disparity is the average disparity value of the height-limited target;

[0026] (4.2) performing multi-step and multi-scale fine-grained filtering with a filtering step number K and a filtering step length S on the data output from the rough cleaning in step (4.1);

[0027] (4.3) calculating the weighted mean of the non-zero disparity values ​​of all rows of the disparity sequence data of the detection area after fine-grained filtering in step (4.2) to obtain the disparity value of the height-limited target;

[0028] (4.4) locating the pixel coordinate information of the lowest edge of the height-limited target in the detection area in the fine-grained filtered disparity data obtained in step (4.2);

[0029] Step 5, using the parallax information of the height-limited target in the detection area obtained in step 4 and the pixel coordinate information of the lowest edge of the height-limited target to calculate the distance measurement value of the height-limited target from the binocular camera and the height measurement value of the height-limited target;

[0030] (5.1) Calculate the actual distance measurement value of the height-limited target from the binocular camera using the internal parameters of the binocular camera and the parallax value of the height-limited target calculated in step (4.3);

[0031] (5.2) Calculate the true height of the height-limited facility from the optical center of the binocular camera using the binocular camera internal parameters, the parallax value of the height-limited target calculated in step (4.3), and the pixel coordinate information of the lowest edge of the height-limited facility obtained in step (4.4), and then calculate the true height measurement value of the height-limited facility from the ground in combination with the installation height of the camera;

[0032]

[0033] Y R =Y+Y CLH

[0034] Among them, Y is the detection height of the height-limited facility from the optical center of the camera, Y CLH is the installation height of the camera, Y R is the actual height of the height-limited facility from the ground, and y is the pixel distance from the optical center of the camera to the bottom edge of the height-limited device;

[0035] Step 6, considering the uncertainty of the data, the distance information between the height-restricted facility and the binocular camera calculated in step 5 is judged to be reasonable and output;

[0036] (6.1) Assuming that the speed change is as reasonable as possible, that is, the distance difference is uniform, the distance measurement value obtained in step (5.1) is subjected to singularity judgment. When the absolute difference between the distance measurement value and the distance prediction value of the current frame obtained based on a priori reasoning is less than the distance difference threshold between two consecutive frames, the distance prediction value is directly output as the final display value. Otherwise, the singularity judgment is not satisfied, and the distance prediction value is optimized and output using the local support and global support methods based on distance difference.

[0037] (6.2) Optimize the distance measurement values ​​that do not meet the singularity judgment in step (6.1), use the global distance control parameter to calculate the global distance prediction value of the current frame, and if the current frame detects frame loss, use the local support polynomial fitting regression method based on the distance difference local support mean filter estimation method to jointly fit the path to obtain the local distance prediction value; if continuous frame loss occurs during the detection, causing the local support polynomial fitting regression method and the local support mean filter method to fail, then use the Kalman filter-based strategy to fit the local path to obtain the local distance prediction value;

[0038] (6.3) Using the distance difference value calculated from the continuous frames and the distance fitting regression error of the current frame using the local support polynomial regression error as evaluation parameters, the global distance prediction value and the local distance prediction value obtained in step (6.2) are evaluated together, and the final prediction value of the distance between the height-restricted facility and the binocular camera is output when the distance measurement value does not meet the singularity judgment condition;

[0039] Step 7, considering the uncertainty of the data, the height measurement value of the height-restricted facility calculated in step 5 is judged and output;

[0040] (7.1) The parallax validity is determined by the distance difference between the continuous frame data, and the height measurement value calculated in step (5.2) is also determined to be valid. When the parallax value and the height measurement data of the current frame data are both determined to be invalid, the height measurement value of the previous frame is output; when the parallax value or the height measurement value is valid, in order to ensure the robustness of the final height output, the height measurement value is optimized and output by using the local support and global support methods based on the distance difference;

[0041] (7.2) Using global control parameters, the global support estimated height is calculated for the height measurement value to be optimized in step (7.1). If the current frame detection does not have frame loss, the height measurement value is filtered by local support weighted mean filtering under the control of local parameters to obtain the averaged local support estimated height. If the current frame detection has frame loss, the Kalman filter is used to fit the local path to reduce the influence of noise, and the obtained local predicted height value is used as the averaged local support estimated height.

[0042] (7.3) Calculate the predicted height boundary value of the height-limited facility in the current scene based on the currently predicted distance between the height-limited target and the binocular camera, the installation height of the binocular camera, and the field of view and focal length of the binocular camera; for safety reasons, take the minimum value of the global support estimated height value and the local support estimated height value of the height-limited facility obtained in step (7.2) as the output; when the minimum value of the global support estimated height value and the local support estimated height value does not meet the height boundary condition, output the default reasonable height, otherwise output the final predicted height of the height-limited facility from the ground;

[0043] Step 8: Visually output the distance measurement value and height measurement value obtained in step 6 and step 7 on the height limit detection interface.

[0044] Furthermore, the target detection network is a Yolo series target detection network or a Faster RCNN target detection network.

[0045] Beneficial effects of the present invention: Due to the diversity and non-standard characteristics of highway height limit equipment, there is currently no open source height limit data set available in this field. The present invention not only solves the problem of difficult detection of height limit equipment, but also ensures that the category information, distance information and height information of the height limit facility are output in real time under the safe braking distance, and the errors of the real-time output distance information and height information are controlled within a reasonable range. The present invention coordinates the rationality judgment of the detection distance and calculated height of the height limit facility through the local support and global support methods, and outputs the distance of the height limit target from the binocular camera and the height of the height limit equipment from the ground in real time, so that the driver can make timely adjustments and reduce safety accidents caused by prediction errors.

[0046] (1) By building a binocular height limit automatic detection experimental platform, collecting a variety of height limit facility scene data to construct a binocular height limit data set, training a height limit target detector, and realizing the binocular height limit target automatic detection function. At the same time, the accuracy of height limit target detection is improved by optimizing the labeling scheme, adjusting the height limit data distribution, adding negative samples to the training set, and other strategies;

[0047] (2) The present invention uses a filtering algorithm for parallax data to effectively filter out outlier noise, thereby improving the accuracy of distance and height measurement of height-restricted targets; if the distance and height measurement method based on sequence images does not make a reasonable judgment on the calculated distance and height of the height-restricted target for each frame of sequence image data, it will cause the height-restricted target to have unreasonable and unstable distance and height output effects in scenes where the distance to the binocular camera is getting closer and closer, and even mislead the driver to make an incorrect judgment. Therefore, the present invention has an important significance for adopting a judgment strategy at the output end, which can provide a reliable reference for the driver when passing through the height-restricted section. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the present invention.

[0049] Figure 2 Schematic diagram of the binocular height-limited scene experimental platform.

[0050] Figure 3 This is the disparity effect diagram of the detection area before filtering.

[0051] Figure 4 This is the parallax effect of the detection area after coarse cleaning and fine-grained filtering.

[0052] Figure 5 This is the flow chart of the distance output strategy.

[0053] Figure 6 It is a high output strategy flow chart. DETAILED DESCRIPTION

[0054] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 The present invention provides an automatic detection method for highway height restriction facilities based on binocular vision sensors. The binocular sensor is used to obtain highway sequence image height restriction scene data. After a trained height restriction target detector is used, the position information of the detection area is output. The disparity data provided by the binocular camera application programming interface is combined to extract the disparity data of the detection area, which is subjected to coarse cleaning and fine-grained filtering. The distance between the height restriction target and the binocular camera and the height of the height restriction target from the optical center of the binocular camera are calculated. After reasonableness judgment, the distance between the height restriction target and the camera and the height of the height restriction target from the ground are output in real time.

[0056] The method for automatically detecting highway height restriction facilities based on binocular vision sensors mainly comprises the following steps:

[0057] Step 1 specifically includes the following sub-steps:

[0058] (1.1) Experimental platform, please refer to Figure 2 , collect a variety of highway binocular height limit scene data, height limit facilities include height limit poles, bridge holes, archways, and tunnels;

[0059] (1.2) Classify highway binocular height restriction scene data according to different height restriction facilities and different appearance features of similar height restriction facilities, add negative sample data with appearance features similar to those of height restriction facilities to the classified data, and obtain highway binocular height restriction scene data;

[0060] (1.3) Using annotation software to annotate the highway binocular height restriction scene data in step (1.2); obtaining a binocular height restriction data set;

[0061] (1.4) Using the application programming interface provided by the binocular camera, obtain the disparity data of the left camera and the right camera after stereo matching in the highway binocular height limit scene data;

[0062] Step 2 specifically includes the following sub-steps:

[0063] (2.1) In view of the insufficient data of height-limited scenes in the highway binocular height-limited dataset, data enhancement is performed, including small-angle rotation, horizontal flipping, blur processing, etc., so that the number of samples of each height-limited scene in the binocular height-limited dataset is evenly distributed;

[0064] (2.2) Input the left camera image and the annotation information corresponding to the left camera image of the data-enhanced binocular height-limited dataset into the selected object detection network for training, such as Yolov5, Faster RCNN, etc., to obtain a height-limited object detector;

[0065] Step 3 specifically includes the following sub-steps:

[0066] (3.1) The left-eye sequence image data of the binocular camera is used as input, and is first processed by the height limit detector obtained in step (2.2), and a two-dimensional detection frame of the detection area of ​​the height limit facility in the sequence image is output, which includes the corresponding position information (x, y, w, h) and the category information of the height limit facility, where x and y represent the pixel coordinates of the center point of the two-dimensional detection frame, and w and h represent the width and height of the two-dimensional detection frame in the image coordinate system respectively;

[0067] (3.2) Using the position information of the two-dimensional detection frame output by step (3.1), the sequence disparity data obtained by step (1.4) is cropped to obtain the disparity sequence data of the detection area; please refer to Figure 3 .;

[0068] (3.1) The left-eye sequence image data of the binocular camera is taken as input, and first processed by the height-limited target detector obtained in step (2.2), and the left-eye sequence image data and the two-dimensional detection frame of the height-limited target area are output, which includes the corresponding position information (x, y, w, h) and the category information of the height-limited facility, where x and y represent the pixel coordinates of the center point of the two-dimensional detection frame, and w and h represent the width and height of the two-dimensional detection frame in the image coordinate system respectively;

[0069] (3.2) Using the position information of the two-dimensional detection frame output by step (3.1) to detect, the disparity data obtained in step (1.4) is clipped to obtain the disparity sequence data of the detection area. In order to avoid the problem of unstable disparity calculation caused by the tilt of the height-limited facility or the tilt of the viewing angle and the excessive length of the height-limited facility at close distance, this scheme adopts a local operator in the target detection area within a certain distance range;

[0070] Step 4 specifically includes the following sub-steps:

[0071] (4.1) The effect of the detection area before coarse-grained filtering can be found in Figure 3, Assumptions: the detection area contains height-limited facilities, the height-limited facilities are horizontal, the height-limited facilities are front features, and the rest are background. In order to achieve lightweight model, only linear operations or convolutions are performed to ensure the real-time performance of the algorithm. The disparity size is inversely proportional to the distance between the height-limited target and the optical center of the camera. According to the internal and external parameters of the binocular camera and the installation height of the camera, the effective working distance of the binocular camera under this scheme and the maximum height of the height-limited facilities that can be detected under extreme conditions can be calculated. Based on the above prior information, the corresponding outlier data threshold lower bound Alpha and outlier data threshold upper bound Beta can be set to roughly clean the disparity sequence data output by step (3.2) to remove invalid disparity values ​​and outliers with large absolute errors between the disparity values ​​and the true disparity values;

[0072]

[0073] Where Z is the detection distance between the height-limited facility and the camera; baseline is the baseline of the binocular camera; f is the focal length of the binocular camera; disparity is the average disparity value of the height-limited target;

[0074] (4.2) The effect of fine-grained filtering in the detection area can be found in Figure 4 , the data outputted by the rough cleaning in step (4.1) is subjected to multi-step and multi-scale fine-grained filtering with a filtering step number K and a filtering step length S. After the rough cleaning operation in step (4.1), the large noise and near-zero disparity data in the disparity data in the detection area are cleaned and replaced with a dynamic non-zero mean, and then multi-step and multi-scale dynamic fine-grained filtering is performed, the fine-grained filtering step number K and step length S are set, and the pixels in the detection area are dynamically fine-grained row by row;

[0075] (4.3) calculating the weighted mean of the non-zero disparity values ​​of all rows of the disparity sequence data of the detection area after fine-grained filtering in step (4.2) to obtain the disparity value of the height-limited target;

[0076] (4.4) After the fine-grained filtering in step (4.2), the disparity between the height-limited target and the background in the detection area is significantly different. According to the assumption in step (4.1), the pixel coordinate information of the bottom edge of the height-limited target in the detection area is located in the data;

[0077] Step 5 specifically includes the following sub-steps:

[0078] (5.1) Calculate the actual distance measurement value of the height-limited target from the binocular camera using the internal parameters of the binocular camera and the parallax value of the height-limited target calculated in step (4.3);

[0079]

[0080] Where Z is the detection distance between the height-limited facility and the camera; baseline is the baseline of the binocular camera; f is the focal length of the binocular camera; disparity is the average disparity value of the height-limited target;

[0081] (5.2) Calculate the true height of the height-limited facility from the optical center of the binocular camera using the binocular camera internal parameters, the parallax value of the height-limited target calculated in step (4.3), and the pixel coordinate information of the lowest edge of the height-limited facility obtained in step (4.4), and then calculate the true height measurement value of the height-limited facility from the ground in combination with the installation height of the camera;

[0082]

[0083] Y R =Y+Y CLH

[0084] Among them, Y is the detection height of the height-limited facility from the optical center of the camera, Y CLH is the installation height of the camera, Y R is the actual height of the height-limited facility from the ground, and y is the pixel distance from the optical center of the camera to the lower edge of the height-limited device.

[0085] Step 6 specifically includes the following sub-steps:

[0086] (6.1) Assuming that the speed change is as reasonable as possible, that is, the distance difference is uniform, the distance measurement value obtained in step (5.1) is subjected to singularity judgment. When the absolute difference between the distance measurement value and the distance prediction value of the current frame obtained based on a priori reasoning is less than the distance difference threshold Diff_i between two consecutive frames, the distance prediction value is directly output as the final display value. Otherwise, the singularity judgment is not satisfied, and the distance prediction value is optimized and output using the local support and global support methods based on distance difference.

[0087] (6.2) Please refer to Figure 5 , optimize the distance measurement value Mea_i that does not meet the singularity judgment in step (6.1), use the global distance control parameter DistanceGlobalPara to calculate the global distance prediction value GlobalEst of the current frame, if the current frame is detected as lost, use the local support polynomial fitting regression method based on the distance difference local support mean filter estimation method to jointly fit the path to obtain the local distance prediction value LocalEst. If continuous frame loss occurs during the detection, causing both the local support polynomial fitting regression method and the local support mean filter method to fail, considering that the Kalman filter can ensure that each frame of the sequence image has a prediction value output, the Kalman filter strategy based on the local control parameter DistanceLocalPara is used to fit the local path to obtain the local distance prediction value LocalEst;

[0088] (6.3) The distance difference value calculated from the continuous frames and the distance fitting regression error Delta_i of the current frame using the local support polynomial are used as evaluation parameters to jointly evaluate the global distance prediction value GlobalEst and the local distance prediction value LocalEst obtained in step (6.2). The distance difference threshold Diff_i can effectively detect and shield the situation where the speed suddenly changes greatly. The GlobalEst and LocalEst are calculated respectively with respect to the distance prediction value EstiHeight_ i-1 Absolute value difference Δ Glo and Δ Loc , the smaller error value between GlobalEst and LocalEst is taken as the final predicted value FinalDisEst of the distance between the height-restricted facility and the binocular camera when the distance measurement value Mea_i does not meet the singularity judgment condition;

[0089] Step 7 specifically includes the following sub-steps:

[0090] (7.1) Please refer to Figure 6 , the parallax validity is determined by the distance difference threshold Diff_i between consecutive frame data, and the height measurement value MeaHeight_i calculated in step (5.2) is also determined to be valid. When the parallax value and height measurement data of the current frame data are both determined to be invalid, the height measurement value EstiHeight_i of the previous frame is used. i-1 Output; When the disparity value or height measurement value is valid, in order to ensure the robustness of the final height output, the height measurement value is optimized and output using the local support and global support methods based on distance difference;

[0091] (7.2) Use the global control parameter HeightGlobalPara to calculate the averaged global support estimated height GlobalHeight_i for the height measurement value that needs to be optimized in step (7.1). If there is no frame loss in the current frame detection, use the local support weighted mean filter to filter the height measurement value under the control of the local parameter HeightLocalPara to obtain the averaged local support estimated height LocalHeight_i. If there is a frame loss in the current frame detection, use the Kalman filter to fit the local path to reduce the influence of noise, and use the obtained local predicted height value as the averaged local support estimated height LocalHeight_i;

[0092] (7.3) Calculate the predicted height boundary value of the height-limited facility in the current scene based on the currently predicted distance of the height-limited target from the binocular camera, the installation height of the binocular camera, and the field of view and focal length of the binocular camera. Since the height calculation depends on the parallax of the effective pixels of the target to be measured, and the closer the distance, the greater the parallax and the higher the accuracy, this scheme adopts the method of approximate cumulative local Bayesian estimation to average the effective values ​​of the local neighborhood. For safety reasons, the weighted average estimation of the effective range is combined with the small-amplitude weighted mean estimation to output the height in a smaller range as much as possible, and the minimum value of the global support estimated height value GlobalHeight_i and the local support estimated height value LocalHeight_i of the height-limited facility obtained in step (7.2) is used as the output. When the minimum value of the global support estimated height value and the local support estimated height value does not meet the height boundary condition, the default reasonable height is used as the final predicted height, otherwise the final predicted height of the height-limited facility from the ground FinalHeiEst is output;

[0093] Step 8: Visually output the distance information FinalDisRes and height information FinalHeiRes obtained in steps 6 and 7 on the height limit detection interface.

Claims

1. A method for automatic detection of highway height restriction facilities based on binocular vision, characterized in that: The steps include: Step 1: Collect binocular height-limited scene data, complete data annotation, and construct a binocular height-limited dataset; (1.1) Collect diverse highway binocular height restriction scene data, including height restriction facilities such as height restriction poles, bridge holes, archways, tunnels, etc. (1.2) Classify highway binocular height restriction scene data according to different height restriction facilities and different appearance features of similar height restriction facilities, add negative sample data with appearance features similar to those of height restriction facilities to the classified data, and obtain highway binocular height restriction scene data; (1.3) Using annotation software to annotate the highway binocular height restriction scene data in step (1.2); obtaining a binocular height restriction data set; (1.4) Using the application programming interface provided by the binocular camera, obtain the disparity data of the left camera and the right camera after stereo matching in the highway binocular height limit scene data; Step 2, using the highway binocular height limit scene data and annotation information in the binocular height limit dataset to train the target detection model and obtain a height limit target detector; (2.1) Perform data enhancement on the highway binocular height limit dataset so that the number of samples of each height limit scene in the binocular height limit dataset is evenly distributed; (2.2) Input the left camera image and the annotation information corresponding to the left camera image of the data-enhanced binocular height-limited dataset into the selected object detection network for training to obtain a height-limited object detector; Step 3: Take the left eye sequence image data of the binocular camera as input, combine it with the disparity data corresponding to the binocular image pair in the binocular height limit data set, and output the disparity information sequence of the detection area of ​​the height limit target as the input of the next stage; (3.1) The left-eye sequence image data of the binocular camera is used as input, and is first processed by the height-limited target detector obtained in step (2.2), and the left-eye sequence image data and the two-dimensional detection frame of the height-limited target area are output, including the position information of the two-dimensional detection frame and the category information of the height-limited facility; (3.2) using the position information of the two-dimensional detection frame output by the detection in step (3.1) to crop the disparity data obtained in step (1.4) to obtain the disparity sequence data of the detection area; Step 4, performing coarse-grained filtering and fine-grained filtering on the disparity sequence data of the detection area obtained in step 3, calculating the disparity value of the height-limited target and locating the pixel coordinate information corresponding to the bottom edge of the height-limited target; (4.1) Since the parallax size is inversely proportional to the distance between the height-limited target and the camera optical center, the lower bound of the outlier data threshold Alpha and the upper bound of the outlier data threshold Betta are set to perform rough cleaning on the parallax sequence data output from step (3.2); Where Z is the detection distance between the height-limited facility and the camera; baseline is the baseline of the binocular camera; f is the focal length of the binocular camera; disparity is the average disparity value of the height-limited target; (4.2) performing multi-step and multi-scale fine-grained filtering with a filtering step number K and a filtering step length S on the data output from the rough cleaning in step (4.1); (4.3) calculating the weighted mean of the non-zero disparity values ​​of all rows of the disparity sequence data of the detection area after fine-grained filtering in step (4.2) to obtain the disparity value of the height-limited target; (4.4) locating the pixel coordinate information of the lowest edge of the height-limited target in the detection area in the fine-grained filtered disparity data obtained in step (4.2); Step 5, using the parallax information of the height-limited target in the detection area obtained in step 4 and the pixel coordinate information of the lowest edge of the height-limited target to calculate the distance measurement value of the height-limited target from the binocular camera and the height measurement value of the height-limited target; (5.1) Calculate the actual distance measurement value of the height-limited target from the binocular camera using the internal parameters of the binocular camera and the parallax value of the height-limited target calculated in step (4.3); (5.2) Calculate the true height of the height-limited facility from the optical center of the binocular camera using the binocular camera internal parameters, the parallax value of the height-limited target calculated in step (4.3), and the pixel coordinate information of the lowest edge of the height-limited facility obtained in step (4.4), and then calculate the true height measurement value of the height-limited facility from the ground in combination with the installation height of the camera; AND R =Y+Y CLH Among them, Y is the detection height of the height-limited facility from the optical center of the camera, Y CLH is the installation height of the camera, Y R is the actual height of the height-limited facility from the ground, and y is the pixel distance from the optical center of the camera to the bottom edge of the height-limited device; Step 6, considering the uncertainty of the data, the distance information between the height-restricted facility and the binocular camera calculated in step 5 is judged to be reasonable and output; (6.1) Assuming that the speed change is as reasonable as possible, that is, the distance difference is uniform, the distance measurement value obtained in step (5.1) is subjected to singularity judgment. When the absolute difference between the distance measurement value and the distance prediction value of the current frame obtained based on a priori reasoning is less than the distance difference threshold between two consecutive frames, the distance prediction value is directly output as the final display value. Otherwise, the singularity judgment is not satisfied, and the distance prediction value is optimized and output using the local support and global support methods based on distance difference. (6.2) Optimize the distance measurement values ​​that do not meet the singularity judgment in step (6.1), use the global distance control parameter to calculate the global distance prediction value of the current frame, and if the current frame detects frame loss, use the local support polynomial fitting regression method based on the distance difference local support mean filter estimation method to jointly fit the path to obtain the local distance prediction value; if continuous frame loss occurs during the detection, causing the local support polynomial fitting regression method and the local support mean filter method to fail, then use the Kalman filter-based strategy to fit the local path to obtain the local distance prediction value; (6.3) Using the distance difference value calculated from the continuous frames and the distance fitting regression error of the current frame using the local support polynomial regression error as evaluation parameters, the global distance prediction value and the local distance prediction value obtained in step (6.2) are evaluated together, and the final prediction value of the distance between the height-restricted facility and the binocular camera is output when the distance measurement value does not meet the singularity judgment condition; Step 7, considering the uncertainty of the data, the height measurement value of the height-restricted facility calculated in step 5 is judged and output; (7.1) The parallax validity is determined by the distance difference between the continuous frame data, and the height measurement value calculated in step (5.2) is also determined to be valid. When the parallax value and the height measurement data of the current frame data are both determined to be invalid, the height measurement value of the previous frame is output; when the parallax value or the height measurement value is valid, in order to ensure the robustness of the final height output, the height measurement value is optimized and output by using the local support and global support methods based on the distance difference; (7.2) Using global control parameters, the global support estimated height is calculated for the height measurement value to be optimized in step (7.1). If the current frame detection does not have frame loss, the height measurement value is filtered by local support weighted mean filtering under the control of local parameters to obtain the averaged local support estimated height. If the current frame detection has frame loss, the Kalman filter is used to fit the local path to reduce the influence of noise, and the obtained local predicted height value is used as the averaged local support estimated height. (7.3) Calculate the predicted height boundary value of the height-limited facility in the current scene based on the currently predicted distance of the height-limited target from the binocular camera, the installation height of the binocular camera, and the field of view and focal length of the binocular camera; for safety reasons, take the minimum value of the global support estimated height value and the local support estimated height value of the height-limited facility obtained in step (7.2) as the output; when the minimum value of the global support estimated height value and the local support estimated height value does not meet the height boundary condition, output the default reasonable height, otherwise output the final predicted height of the height-limited facility from the optical center of the camera; Step 8: Visually output the distance measurement value and height measurement value obtained in step 6 and step 7 on the height limit detection interface.

2. The method for automatic detection of highway height restriction facilities based on binocular vision according to claim 1 is characterized in that: The target detection network is a Yolo series target detection network or a Faster RCNN target detection network.

Images