A method for determining the visual distance of highway video monitoring under scene prior constraints

Abstract

The application designs a highway video monitoring visual distance determination method under scene prior constraint, and belongs to the field of image processing.The method realizes automatic learning of highway structured scene information, and calibrates a camera depth scale based on road prior and vehicle model prior, constructs a visual distance calculation model based on three criteria of imaging area historical background information and current visible edge contrast, farthest target recognition vehicle position and historical background information correction transmissivity graph, and calculates real-time visual distance of the camera on the current erected road.The visual distance estimation method based on ordinary monocular monitoring camera can be widely applied to the monitoring cameras erected on various road sections of the highway, and can roughly determine the visual distance of the current road section without professional equipment and without knowing other detailed data of the camera, thereby helping to realize intelligent transportation.

Description

Technical Field

[0001] This invention relates to a method for determining the line-of-sight distance of highway video surveillance under scenario prior constraints. Background Technology

[0002] In recent years, the importance of traffic safety has been increasing. However, the distribution of fog on different road sections often varies in foggy weather. With the advancement of intelligent transportation systems, real-time monitoring of the visibility distance of various road sections over a wide area can greatly improve the traffic management department's ability to control road safety in foggy weather.

[0003] While current methods for detecting road visibility distance are relatively mature, the following problems still exist:

[0004] (1) Current visibility measuring instruments are difficult to popularize.

[0005] Currently, the common method for detecting road visibility distance is to install expensive professional meteorological equipment, which has the advantage of high detection accuracy. However, this method is costly and can often only be applied to key road sections, making it impossible to achieve large-scale, widespread detection.

[0006] (2) Current road visibility detection algorithms rely on a single criterion for determining visibility distance.

[0007] Current road visibility detection algorithms often rely on a single principle to obtain the visibility distance result. Their judgment is mostly based on a certain inference of the meteorological definition of visibility. However, such visibility detection algorithms based on a single criterion will fail in certain specific scenarios, and the general applicability of the algorithm is poor.

[0008] (3) Current road visibility detection algorithms do not fully utilize the structured scene information and scene priors of highways.

[0009] Given the abundance of structured information available in the application scenario of highways and the specific equipment of road surveillance cameras, most current algorithms have not been specifically optimized for this scenario. Summary of the Invention

[0010] To address the aforementioned problems in existing systems, this invention provides a method for determining the visible distance of highway video surveillance under scene prior constraints, which achieves the following effects: 1) It realizes automatic learning of structured scene information of highways and automatic calibration of camera depth scales; 2) It can make full use of the prior information of highway scenes, including structured information, historical background information and vehicle model information; 3) It can adaptively calculate the final visible distance estimation result based on multiple visible distance judgment benchmarks.

[0011] According to one aspect of the present invention, a method for determining the line-of-sight distance of highway video surveillance under scenario prior constraints is provided, characterized by comprising the following steps:

[0012] A) Automatically learns structured scene information of highways and generates multi-frame fused background images based on long-term video streams from the current fixed viewpoint of the camera.

[0013] B) Based on highway scene priors and vehicle model priors, the actual distance transformation scale of the monocular camera image is calibrated;

[0014] C) Based on the clear background edge map obtained by multi-frame fusion, extract edge information from the real-time imaging image under the current weather conditions and compare it with the background edge information to obtain the coordinates of the invisible edge points under the current weather conditions. Substitute the above coordinates into the actual distance transformation scale of the monocular camera imaging image to obtain the first visible distance benchmark.

[0015]

[0016] Where L1 is the first visible distance reference, λ is the camera projection model calibration parameter, and diff y The row coordinates of the farthest edge position are obtained by comparing the background edge map with the currently visible edge information.

[0017] D) Calculate the coordinates of the farthest vehicle that the vehicle recognition model can detect in recent imaging images, and substitute these coordinates into the actual distance transformation scale of the monocular camera imaging image to obtain the second visible distance benchmark.

[0018]

[0019] Where L2 is the second visible distance reference, box y The row coordinates of the upper edge of the target detection box are determined.

[0020] E) A transmittance calculation algorithm based on color ellipsoid prior is used to calculate the transmittance of the camera's image to obtain a real-time transmittance map. Based on the historical scene information collected in step A), the transmittance map is corrected to remove transmittance information strongly correlated with the colors of man-made objects in the scene. Then, the transmittance map is adaptively divided into regions to obtain the coordinates of the lower boundary of the high transmittance region. These coordinates are then substituted into the actual distance transformation scale of the monocular camera's image to obtain a third visible distance reference.

[0021]

[0022] L3 is the third line-of-sight distance benchmark, roi y These are the row coordinates of the lower edge of the high-transmittance region obtained after region segmentation.

[0023] F) The algorithm intelligently senses the current fog concentration and, under three conditions—clear sky, light fog, foggy, and dense fog—substitutes the three visibility distance benchmarks obtained in steps C), D), and E) into different calculation models to obtain the final estimated visibility distance value. For scenarios where the calculated visibility distance is greater than 500m, the algorithm outputs the visibility distance based on the background comparison result. out =L1. When the fog is dense but does not completely cover the camera's image, L... out = L1 + 0.4 * min{|L2 - L1|, |L3 - L1|}. When the entire image captured by the camera is covered by fog in dense fog, the algorithm output is: L out = (L2 + L3) / 2; Attached Figure Description

[0024] Figure 1 A background image is generated by multi-frame fusion according to an embodiment of the present invention.

[0025] Figure 2(A) is a historical background information map according to an embodiment of the present invention, and Figure 2(B) is a current real-time screen edge map.

[0026] Figure 3(A) is a transmittance diagram before adjustment according to an embodiment of the present invention, and Figure 3(B) is a transmittance diagram after primary color adjustment and region segmentation. Detailed Implementation

[0027] A method for determining the line-of-sight distance of highway video surveillance under scenario prior constraints according to an embodiment of the present invention includes the following steps:

[0028] a) Generate a multi-frame fused background image based on the long-term video stream from the current fixed viewpoint of the camera to extract structured information;

[0029] b) Based on the prior knowledge of the structured scene of the highway and the prior knowledge of the vehicle model, the depth-of-field scale of the camera is calibrated;

[0030] c) Obtain the first visible distance benchmark based on the historical background information of the imaging area and the comparison of the current visible edges;

[0031] d) Statistically analyze the coordinates of the farthest vehicle obtained from vehicle identification to obtain the second visible distance benchmark;

[0032] e) Based on historical scene information, the transmittance map is corrected and the coordinates of the lower boundary of the high transmittance area are obtained to obtain the third visible distance reference.

[0033] f) Substitute the first to third visual distance benchmarks into the final visual distance calculation model to obtain the final estimated visual distance value.

[0034] Specific implementation examples of the above operations will now be described.

[0035] a) Background image generation

[0036] Cameras installed on highways typically capture images from a fixed perspective for extended periods, allowing background information to be extracted using long-exposure algorithms that fuse multiple frames.

[0037] During long exposure, edge information is continuously extracted from the image, and a binarization threshold is periodically set to remove ghosting left by moving targets in the image, ultimately resulting in a road scene edge background image with less interference information. An example of the edge background image is shown below. Figure 1 .

[0038] b) Determining the camera depth-of-field scale

[0039] According to the projection relationship, for any row coordinate v in the image, its actual distance d from the origin can be expressed as:

[0040]

[0041] Among them, v p The vanishing point of the image is represented by the row coordinates λ, which can be determined by a pair of pixels with known actual distances:

[0042]

[0043] Where d1-d2 is the actual relative distance between a pair of pixels, and v1 and v2 are the row coordinates of the pair of pixels respectively.

[0044] The above model applies when the camera's optical axis is collinear with the direction of the road in the image. If the camera is tilted to the left or right, corresponding corrections are required. Furthermore, due to the camera's downward angle, the calculated distance is slightly longer than the actual distance directly below the camera. For example, with a camera height of 10m and a downward angle of 15°, the calculated distance is approximately 2.5m longer.

[0045] The camera projection model requires the vanishing point coordinates to be obtained before the actual distance can be calculated.

[0046] The vanishing point coordinates are obtained by detecting road edges and lane lines in the background image and finding the intersection of the equations of any two straight lines. Different line segments and detection results from different frames can be used to calibrate the vanishing point position.

[0047] To obtain a camera distance scale, a reference object with a known actual length in the image is needed. The dashed lane line segment is more accurate than the vehicle length detection box obtained from target detection. According to national highway lane line specifications, the length of a dashed lane line segment is 6m, and the interval between dashed segments is 9m.

[0048] A line segment detection model is used to identify the edges of parallel lane lines of similar length and within 10 pixels of each other. Lane lines are then detected on background images generated based on long-term historical information to improve detection accuracy. More accurate camera calibration parameters can be obtained on roads with suitable scenes and clearly defined lane dashed lines.

[0049] c) Obtain the first visible distance benchmark based on the historical background information of the imaging area and the comparison of the current visible edges;

[0050] Fixed highway scene background information is extracted and saved from a long-term video stream. Edge information in real-time footage under foggy or other weather conditions is extracted and compared with the saved information. The position scale corresponding to the lower edge of the difference between the two can be used as the current visible distance. An example of the real-time image edge is shown in Figure 2(A), which is compared with the edge background image shown in Figure 2(B).

[0051]

[0052] Where L1 is the first visible distance reference, λ is the camera projection model calibration parameter, and diff y The row coordinates of the farthest edge position are obtained by comparing the background edge map with the currently visible edge information.

[0053] d) Statistically analyze the coordinates of the farthest vehicle obtained from vehicle identification to obtain the second line-of-sight distance benchmark.

[0054] The visual distance detection result is output by averaging the positions of the farthest vehicles that the target detection model can detect within a certain frame sliding time window.

[0055]

[0056] Where L2 is the second visible distance reference, box y The row coordinates of the upper edge of the target detection box are determined.

[0057] e) Correct the transmittance map based on historical scene information and obtain the coordinates of the lower boundary of the high transmittance area to obtain the third visible distance reference.

[0058] An unadjusted transmittance map contains numerous transmittance variations strongly correlated with scene colors. Historical scene primary color information can be used to adjust the transmittance map, eliminating transmittance variations with low correlation to fog information. Based on the adjusted transmittance map, region segmentation is performed to obtain the lower edge position of distant high-transmittance areas and use it as the visible distance value. An example of the transmittance map before adjustment is shown in Figure 3(A), and the effect after transmittance primary color adjustment and region segmentation is shown in Figure 3(B).

[0059]

[0060] L3 is the third line-of-sight distance benchmark, roi y These are the row coordinates of the lower edge of the high-transmittance region obtained after region segmentation.

[0061] f) Obtain the final estimated visual distance value

[0062] In clear or lightly foggy conditions, real-time footage can detect edge information at almost the same distance as the background. In this case, the visibility distance output is primarily based on the background comparison benchmark. However, in scenarios where the visibility distance is greater than 500m, the target detection algorithm performs poorly on distant vehicle targets due to the viewing angle, making it difficult to output stable results. Therefore, the algorithm outputs the visibility distance based on the background comparison result. out =L1.

[0063] Even when the fog is thick but does not cover the entire camera image, meaning that a lot of edge information can still be extracted from the real-time image, the comparison result between the current image and the background information can still be used as the main basis for the output of the visible distance (usually less than 500m). At the same time, the output result is adjusted in real time based on the farthest vehicle target feedback from the target detection.

[0064] The final algorithm outputs a visible distance value of L. out =L1+0.4*min{|L2-L1|, |L3-L1|}.

[0065] When the entire camera image is covered by fog in dense fog, the edge information in the real-time image is weak. Edge extraction methods using a fixed extraction threshold are almost ineffective (the number of non-zero pixels in the real-time edge image is less than the number of non-zero pixels in the background image * 0.1). In this situation, the visible distance result obtained from the background comparison benchmark is essentially invalid. The algorithm output is: L out = (L2+L3) / 2.

[0066] Advantages and beneficial effects of the invention:

[0067] This invention enables automatic learning of structured scene information on highways and obtains a camera depth scale based on road priors and vehicle type prior calibration. It constructs a visible distance calculation model based on three benchmarks: historical background information of the imaging area and comparison of currently visible edges; vehicle position identification based on the farthest target; and transmittance map correction based on historical background information. This model calculates the real-time visible distance of the camera on the currently mounted road. This invention can be applied to estimating the visible distance of ordinary monocular surveillance cameras on highways and can be widely used with existing surveillance cameras installed on various road sections. It can roughly determine the visible distance of the current road section without requiring specialized equipment or other detailed camera data, thus contributing to the realization of intelligent transportation.

Claims

1. A method for determining the visual distance of highway video monitoring under the scenario prior constraint, characterized in that Comprising the following steps: A) generating a multi-frame fused background picture according to long-term video stream under the current fixed view angle of the camera; B) calibrating the actual distance conversion scale of the monocular camera imaging picture based on the highway scene priori and vehicle model priori; C) extracting edge information from the real-time imaging picture under the current weather and comparing it with the background edge information according to the clear background edge map obtained by multi-frame fusion, obtaining the edge invisible point coordinates of the current weather in real time, and substituting the coordinates of the edge invisible point into the actual distance conversion scale of the monocular camera imaging picture to obtain the first visible distance reference; wherein L1 is a first visual distance reference, v p is the line coordinate of the vanishing point of the imaging picture, λ is the calibration parameter of the camera projection model, diff y is the line coordinate of the farthest edge position obtained by comparing the background edge map with the current visible edge information, D) substituting the coordinates of the farthest vehicle position that can be detected by the vehicle recognition model in the recent imaging picture into the actual distance conversion scale of the monocular camera imaging picture to obtain the second visible distance reference; wherein L2 is a second visual distance reference, box y is a row coordinate of the upper edge of the target recognition detection box. E) using the transmittance calculation algorithm based on the color ellipsoid priori to obtain the real-time transmittance map, and correcting the real-time transmittance map based on the long-term video stream collected in step A) to remove the transmittance information strongly related to the color of the scene artificial object, and then performing adaptive region division on the real-time transmittance map to obtain the coordinates of the lower boundary of the high transmittance region, and substituting the coordinates of the lower boundary of the high transmittance region into the actual distance conversion scale of the monocular camera imaging picture to obtain the third visible distance reference; wherein L3 is a third visual distance reference, roi y is a lower edge row coordinate value of the high-transmittance region obtained after region segmentation, F) determining the visible distance value according to the current weather fog concentration, comprising: In the scene where the visual distance measurement value is greater than 500m, take L out = L1, L = L1 + 0.4 * min{|L2 - L1|, |L3 - L1|} out = L1 + 0.4 * min{|L2 - L1|, |L3 - L1|}, In the case of the whole imaging picture of the camera is covered by fog in thick fog weather, take L out = (L2+L3) / 2, Wherein: Step B) comprises: According to the projection relationship, the actual distance d of any one row coordinate v in the imaging picture and the coordinate origin is expressed as: where v p is the vanishing point row coordinate of the imaged picture, and λ is calibrated from a pair of pixel points with known actual distances. Wherein, d1-d2 is the actual relative distance of a pair of pixel points, and v1, v2 are the respective row coordinates of the pair of pixel points.

2. A computer-readable storage medium storing a computer program, which can enable a processor to execute the method according to claim 1.

3. A highway visible distance estimation device characterized by Comprising: A multi-frame fused background picture generation part for generating a multi-frame fused background picture according to long-term video stream under the current fixed view angle of the camera; A calibration part for calibrating the actual distance conversion scale of the monocular camera imaging picture based on the highway scene priori and vehicle model priori; A first visible distance reference determination part for extracting edge information from the real-time imaging picture under the current weather and comparing it with the background edge information according to the clear background edge map obtained by multi-frame fusion, obtaining the edge invisible point coordinates of the current weather in real time, and substituting the coordinates of the edge invisible point into the actual distance conversion scale of the monocular camera imaging picture to obtain the first visible distance reference; wherein L1 is a first visual distance reference, v p is the farthest edge position row coordinate obtained by comparing the background edge map with the current visible edge information, and y is the farthest edge position row coordinate obtained by comparing the background edge map with the current visible edge information, and A second visible distance reference determination part for substituting the coordinates of the farthest vehicle position that can be detected by the vehicle recognition model in the recent imaging picture into the actual distance conversion scale of the monocular camera imaging picture to obtain the second visible distance reference; wherein L2 is a second visual distance reference, box y is a row coordinate of the upper edge of the target recognition detection box, A third visual distance reference determining part is configured to obtain a real-time transmittance map using a transmittance calculation algorithm based on color ellipsoid prior for the imaging picture of the camera, and correct the transmittance map based on the long-term video stream collected in step A) to remove the transmittance information strongly related to the color of the artificial object in the scene, and then perform adaptive region division on the transmittance map to obtain the coordinates of the lower boundary of the high transmittance region, and substitute the coordinates of the lower boundary of the high transmittance region into the actual distance conversion scale of the monocular camera imaging picture to obtain a third visual distance reference; wherein L3 is a third visual distance reference, roi y is a lower edge row coordinate value of the high transmittance region obtained after region segmentation, A visual distance value determining part is configured to: In the scene where the visual distance measurement value is greater than 500m, take L out = L1, L = L1 + 0.4 * min{|L2 - L1|, |L3 - L1|} out = L1 + 0.4 * min{|L2 - L1|, |L3 - L1|}, In the case of the whole imaging picture of the camera is covered by fog in thick fog weather, take L out = (L2+L3) / 2, Wherein: The calibration part is configured to express the actual distance d of any one row coordinate v in the imaging picture from the coordinate origin as: where v p is the vanishing point row coordinate of the imaged picture, and λ is calibrated from a pair of pixel points with known actual distances. Wherein, d1-d2 is the actual relative distance of a pair of pixel points, and v1 and v2 are the respective row coordinates of the pair of pixel points.

Images