A method and apparatus for atmospheric visibility estimation based on the fusion of binocular stereo vision and deep learning

Abstract

This invention discloses a visibility estimation method and apparatus based on binocular vision. The method includes: simultaneously acquiring left and right views using a calibrated binocular camera; inputting the image pairs into a stereo matching deep neural network to obtain a disparity map and converting it into a depth map; inputting the left view and depth map into a dual-branch deep convolutional neural network to extract multi-scale features; fusing RGB features and depth features through a cross-modal feature fusion module; and finally outputting a visibility estimate through a feature aggregation and regression module. The apparatus includes a binocular image acquisition unit, a data processing and visibility estimation unit, and a result output unit. This invention solves the depth ambiguity problem of monocular vision by fusing binocular depth information and image appearance information, achieving high-precision, non-contact visibility surface measurement. It has the advantages of low cost and flexible deployment, and is suitable for visibility monitoring in traffic scenarios such as highways.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, meteorological monitoring and traffic safety technology, and specifically to an atmospheric visibility estimation method and device based on the fusion of binocular stereo vision and deep learning. Background Technology

[0002] Atmospheric visibility, usually referring to meteorological optical range, is an important physical quantity characterizing atmospheric transparency and reflecting the intensity of severe weather such as fog, haze, rain, and snow. Accurate and real-time monitoring of this visibility is irreplaceable for ensuring transportation safety, guiding aviation and maritime operations, assessing environmental pollution, and military deployment. Particularly in highway systems, localized fog patches and sudden dense fog are major contributing factors to catastrophic chain-reaction rear-end collisions. Therefore, developing a visibility monitoring technology that can achieve wide-area, real-time, high-precision, and cost-effective monitoring has become an urgent need in the field of traffic meteorology. Currently, mainstream visibility measurement technologies are mainly divided into two categories: contact measurement and remote sensing measurement, each with significant technical limitations.

[0003] A representative contact-based point measurement technique is the forward-scattering visibility meter. Its working principle involves emitting a beam of light of a specific wavelength and measuring the intensity of the light scattered forward at a certain angle, typically ranging from 30 to 45 degrees. The extinction coefficient is then derived using Mie scattering theory, and the meteorological optical range is calculated accordingly. This method offers high measurement accuracy and has been adopted as the benchmark measurement method by the World Meteorological Organization.

[0004] However, its inherent drawback lies in its ability to measure atmospheric conditions at a single point, namely the location of the instrument probe, making it a "contact" measurement. This means the technology lacks spatial representativeness: measurements from a single point cannot effectively reflect the spatial distribution differences in visibility over a distance of several kilometers, and it is highly susceptible to missing spatial inhomogeneities such as fog patches. Furthermore, it lacks telemetry capabilities, making it unable to perceive atmospheric conditions in distant or inaccessible areas, such as bridges, tunnel exits, or canyons. In addition, deployment and maintenance costs are high: dense deployment along roads is required to achieve limited spatial coverage, leading to a significant increase in the costs of equipment procurement, installation, power supply, and long-term maintenance.

[0005] Active remote sensing technology is mainly represented by lidar. Lidar emits laser pulses into the atmosphere and receives the backscattered signals. By analyzing the signal decay over time, it can retrieve the spatial profile of the extinction coefficient, thereby achieving remote sensing measurements within a certain range. Although its technology is advanced and can provide more dimensions of atmospheric information, its inherent limitations restrict its large-scale application.

[0006] The primary issue is the extremely high cost: the core laser and sophisticated optical system are expensive, making the cost of a single unit far higher than that of a forward scattering instrument. Secondly, it suffers from poor environmental adaptability: the complex optical system is sensitive to vibration and changes in temperature and humidity, posing challenges to its long-term stability and reliability in harsh outdoor weather conditions, and making maintenance complex. Furthermore, there are safety and regulatory concerns: high-power lasers pose a potential risk to human eye safety, which may restrict their deployment in public areas.

[0007] With the increasing prevalence of cameras in traffic monitoring, image analysis-based visibility estimation methods have gained attention due to their potential for non-contact, low-cost, and easy deployment; this falls under the category of passive vision measurement technology. Existing vision methods can be further subdivided.

[0008] Among these methods, landmark-based approaches require the presence of fixed reference objects at known precise distances in the video footage, such as signs, lampposts, or ridgelines. This approach inverses visibility by establishing a physical model relating the contrast, sharpness, distance, and visibility of reference objects in the image; commonly used theoretical foundations include Koschmidt's law. However, its application is severely limited by the scenario: in open sections of road lacking suitable landmarks, on curves, or on newly constructed roads, this method completely fails, exhibiting extremely poor generalization ability.

[0009] To reduce reliance on markers, researchers have explored marker-free, data-driven approaches, using deep learning models to directly regress visibility values ​​from monocular images. These methods typically employ large convolutional neural networks as feature extractors to learn the mapping between global or local image features and visibility.

[0010] Despite some progress, the fundamental scientific problem lies in deep ambiguity: a single 2D image simultaneously carries information about scene content, scene structure, and atmospheric conditions, specifically including object color and texture, object distance, and atmospheric extinction. Networks cannot reliably distinguish between "objects appearing blurry because they are gray" and "objects appearing blurry because of fog" solely from appearance changes. Models often overfit to specific scene layouts and lighting conditions in the training dataset; once deployed to new environments, their accuracy drops sharply due to differences in scene appearance, demonstrating insufficient robustness.

[0011] In summary, current technologies cannot simultaneously meet the four key requirements of high precision, high robustness, low cost, and wide-area telemetry. Binocular stereo vision technology can passively recover dense scene depth information from a pair of images, providing a physical basis for solving the "depth ambiguity" problem in monocular vision. However, how to effectively fuse depth information with image appearance information to construct a unified model that understands "where objects are" and "how objects look," thereby achieving high-precision and highly generalizable estimation of atmospheric extinction effects, remains a largely unresolved technical problem. This invention addresses this technological gap. Summary of the Invention

[0012] The purpose of this invention is to overcome the shortcomings of existing visibility measurement technologies in terms of accuracy, cost, telemetry capability, and scene dependence, and to provide a visibility estimation method and device based on binocular vision. This method creatively integrates the geometric depth information recovered by binocular stereo vision with image color information, and through an end-to-end deep learning framework, achieves accurate, robust, and non-contact surface estimation of atmospheric visibility.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: A visibility estimation method based on binocular vision, characterized by the following steps: S1: Synchronously acquire the left view using a calibrated binocular camera. and right view This constitutes the original image pair.

[0014] S2: Input the original image pair into a pre-trained stereo matching deep neural network to obtain a disparity map as the output. Based on the binocular camera calibration parameters, the disparity map is converted into a depth map representing the absolute distance between each pixel in the scene. .

[0015] S3: Move the left view and the depth map The data are input into a dual-branch deep convolutional neural network for feature extraction, resulting in multi-scale RGB appearance feature groups. and depth geometric feature group .

[0016] S4: Through a cross-modal feature fusion module, RGB appearance features and depth geometric features at the corresponding scale are adaptively fused to generate a fused feature set. .

[0017] S5: Combine the fused feature groups The input is fed into a feature aggregation and regression module, which, through spatial aggregation and nonlinear mapping, outputs the final visibility estimate. .

[0018] In step S2, the stereo matching deep neural network adopts a RAFT-Stereo or CREStereo architecture. (Depth map) The calculation formula is:

[0019] in, Indicates the baseline length of the stereo camera. This indicates the camera's focal length (in pixels). This represents the disparity value predicted by the network.

[0020] In step S3, the dual-branch deep convolutional neural network is constructed using ResNet as the backbone network. The RGB feature extraction branch consists of ResNet modules from the first three stages (stage 1, stage 2, stage 3), with a three-channel input. The deep feature extraction branch structure is the same as the RGB branch, but its input is a single channel. The depth map is normalized and copied into a three-channel format before input to fit the network input. The parameters of the two branches are not shared, allowing them to learn specific representations for color texture and distance information respectively.

[0021] In step S4, the cross-modal feature fusion module employs a fusion strategy based on a bilinear attention mechanism. For the first... Features extracted from layers and First, a spatial attention weight map is generated through a lightweight convolutional attention subnetwork. This sub-network will and After concatenation along the channel dimension, a 1x1 convolution for dimensionality reduction, ReLU activation, and then another 1x1 convolution followed by Sigmoid activation are performed to obtain the final product. The fusion process can be described as follows:

[0022] in, Indicates channel splicing. This represents a 1x1 convolution operation. and These represent the ReLU and Sigmoid activation functions, respectively. This represents element-wise multiplication. This mechanism allows the network to adaptively determine whether to rely more on appearance information or geometric information in different regions of an image.

[0023] In step S5, the feature aggregation and regression module first processes the fused feature map at the final scale. Perform global average pooling to obtain a global feature vector. Subsequently, The data is passed through two fully connected layers, with ReLU activation and Dropout operations in between to prevent overfitting, and finally outputs a scalar value. The process is represented as:

[0024] in, Indicates global average pooling. and Indicates a fully connected layer. This indicates random inactivation.

[0025] The method employs a smoothing loss function during the training phase. loss:

[0026] in, This represents the actual visibility value. This is the smoothing threshold (usually set to 300).

[0027] A visibility estimation device based on binocular vision, used to implement the above method, mainly includes a binocular image acquisition unit, consisting of a binocular camera with a fixed baseline and calibrated parameters, used to simultaneously capture left and right views; a data processing and visibility estimation unit, including a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the program, can implement the various method steps as described above; and a result output unit, which is responsible for displaying, storing, or transmitting the estimated visibility value locally to a remote monitoring center via a network.

[0028] In a preferred embodiment, the binocular camera has a baseline length of 300 mm, a lens diagonal field of view of 45 degrees, and a monocular sensor resolution of 2 megapixels (i.e., 1920 × 1080). This particular configuration achieves a good engineering balance between depth estimation accuracy and field of view coverage within a visibility estimation range of approximately 50 meters to 2000 meters.

[0029] Compared with existing technologies, this invention exhibits several significant advantages. First, it achieves high-precision non-contact surface telemetry. By utilizing the principle of binocular vision, this method can comprehensively estimate the overall visibility within the field of view without deploying any physical equipment in the area to be measured, thereby effectively capturing spatial non-uniform phenomena such as fog.

[0030] Secondly, this method fundamentally solves the depth ambiguity problem inherent in monocular vision methods. By introducing an accurate depth map generated by a stereo matching network, a crucial physical distance benchmark is provided for visibility estimation. The employed bimodal fusion network can clearly distinguish between the inherent properties of the scene and atmospheric attenuation effects, which greatly improves the estimation accuracy of the model and its robustness across different scenes.

[0031] Its innovative network architecture design is also one of its core advantages. The proposed dual-branch feature extraction structure and attention-based feature fusion module can adaptively and efficiently jointly mine the intrinsic correlation between appearance information and depth geometric information, which is the key technology for achieving high-precision estimation.

[0032] Furthermore, this device is characterized by its low cost, ease of deployment, and widespread adoption. Its core hardware consists of an industrial binocular camera and a general-purpose computing device, resulting in an overall cost far lower than that of a lidar system. It can also be easily integrated with existing traffic monitoring systems, making it ideal for large-scale deployment in areas requiring extensive monitoring, such as highways, airports, and ports.

[0033] Finally, this method provides an end-to-end automated process. From the raw binocular image input to the final visibility value output, the entire process requires no manual intervention or pre-defined scene knowledge, resulting in a high degree of automation and strong practicality. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the system composition of the visibility estimation device in an embodiment of the present invention.

[0035] Figure 2 This is a flowchart illustrating the overall process of the visibility estimation method of the present invention.

[0036] Figure 3 This is a detailed structural diagram of the dual-modal fusion deep neural network in this invention.

[0037] Figure 4 This is a schematic diagram illustrating the deployment and imaging geometry of the binocular camera in the embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0039] Example 1: A Visibility Estimation System for Highway Scenarios This embodiment describes a complete system implementation scheme for highway fog warning.

[0040] 1. Hardware system deployment and configuration like Figure 1 and Figure 4 As shown, the device is deployed on a gantry or column along the highway, with an installation height of approximately [missing information]. = 6 meters, camera optical axis tilt angle downwards = 10°, for optimal coverage of multiple lanes and roadside emergency areas. The core sensing unit uses a fixed baseline length of 10°. = 300 mm industrial-grade binocular camera module. The lens has a diagonal field of view (DFOV) of 45°, corresponding to a horizontal field of view of approximately 36°. The camera's two built-in global shutter CMOS sensors each have a resolution of 2 megapixels (1920 pixels × 1080 pixels). This configuration has been carefully optimized: the 300 mm baseline provides sufficient parallax resolution within the target measurement range (50 meters to 2000 meters) to ensure accurate depth estimation; the 45° field of view strikes a balance between covering a sufficiently wide road area and maintaining resolution in the image's central region; and the 2 megapixel resolution provides sufficient texture detail for objects at medium to long distances, supporting reliable operation of stereo matching networks.

[0041] The data processing unit uses the NVIDIA Jetson AGX Orin embedded AI platform. The device connects to the highway private network via Ethernet or a 5G module, uploading the estimation results to the cloud control platform in real time.

[0042] 2. Software Algorithm Implementation Details The software implementation of the entire algorithm includes two stages: offline training and online inference.

[0043] During online inference, the stereo matching depth estimation module will synchronously acquire the left view. and right view The input is fed into a lightweight RAFT-Stereo network, which outputs a disparity map. Subsequently, based on the camera's intrinsic parameters (focal length) , and the main point , ) and baseline Calculate the absolute depth map according to formula (1) To improve the robustness of subsequent processing, the depth map is normalized: ,in and The depths are set to 5 meters and 2000 meters respectively, and depths exceeding these ranges are truncated.

[0044] Regarding the visibility estimation network structure, the end-to-end visibility estimation network structure constructed in this embodiment is as follows: Figure 3 As shown, its input is the left view. and normalized depth map The output is the visibility value. .

[0045] In the dual-branch feature extractor, the RGB branch is based on a ResNet-34 pre-trained on ImageNet, with its last fully connected layer and average pooling layer removed. The first three modules (Stage 1 to Stage 3) are used as the feature extractor. The input is a size [ , The left view of [3] outputs feature maps at three scales, with sizes of 1 / 4, 1 / 8, and 1 / 16 of the original image, respectively, and the number of channels... The values ​​are 64, 128, and 256 respectively. They are denoted as:

Claims

1. A visibility estimation method based on binocular vision, characterized in that, Includes the following steps: S1: Synchronously acquire the left view using a calibrated binocular camera. and right view ; S2: Input the image pair into a pre-trained stereo matching deep neural network to output a disparity map. The disparity map is converted into a depth map based on the binocular camera calibration parameters. ; S3: Move the left view and the depth map The data are input into a dual-branch deep convolutional neural network to extract multi-scale RGB appearance feature groups. and depth geometric feature group ; S4: Through the cross-modal feature fusion module, RGB appearance features and depth geometric features at the corresponding scale are adaptively fused to generate a fused feature set. ; S5: Combine the fused feature groups Input to the feature aggregation and regression module, output visibility estimate .

2. The visibility estimation method based on binocular vision according to claim 1, characterized in that, The stereo matching deep neural network in step S2 is an end-to-end stereo matching deep network.

3. The visibility estimation method based on binocular vision according to claim 1, characterized in that, The depth map in step S2 The calculation formula is: ， in, Indicates the baseline length of the stereo camera. Indicates the camera's focal length. This represents the disparity value predicted by the network.

4. The visibility estimation method based on binocular vision according to claim 1, characterized in that, The dual-branch deep convolutional neural network in step S3 is constructed with ResNet as the backbone network. The RGB feature extraction branch and the deep feature extraction branch have the same structure but do not share parameters.

5. The visibility estimation method based on binocular vision according to claim 1, characterized in that, The cross-modal feature fusion module in step S4 adopts an attention-based fusion strategy, specifically as follows: For the features extracted from the l-th layer and Spatial attention weight map is generated through a convolutional attention subnetwork. The fusion process is represented as: ， in, This indicates element-wise multiplication.

6. The visibility estimation method based on binocular vision according to claim 1, characterized in that, The feature aggregation and regression module in step S5 includes a global average pooling layer and a fully connected layer, specifically: Fusion features Global average pooling is performed to obtain the global feature vector. Then, visibility estimates are obtained through regression using two fully connected layers. .

7. The visibility estimation method based on binocular vision according to claim 1, characterized in that, The training method described above uses smoothing. loss : ， in, This represents the actual visibility value. This is the smoothing threshold (usually set to 300).

8. A visibility estimation device based on binocular vision, used to implement the method described in any one of claims 1-7, characterized in that, include: Binocular image acquisition unit: Consists of a binocular camera with a fixed baseline and calibrated parameters, used to simultaneously capture the left and right views; Data processing and visibility estimation unit: includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method as described in any one of claims 1-7; Result output unit: used to display, store or transmit the estimated visibility value.

9. The visibility estimation device based on binocular vision according to claim 8, characterized in that, The binocular camera has a baseline length of 300mm, a lens diagonal field of view of 45 degrees, and a monocular sensor resolution of 2 megapixels.

10. The visibility estimation device based on binocular vision according to claim 8, characterized in that, The data processing and visibility estimation unit adopts an embedded AI computing platform, and the device is deployed along the highway for traffic visibility monitoring and early warning.

Images