Image depth learning-based visibility monitoring method, system, terminal and medium

Abstract

This invention discloses a visibility monitoring method, system, terminal, and medium based on deep learning for images, relating to the field of computer vision technology. The key technical points are: This invention utilizes the powerful general visual feature extraction capabilities of a ResNet network pre-trained on a massive dataset as a strong baseline for processing visibility images; by freezing its shallow and mid-layer parameters, the universal representation of basic features such as edges and textures is preserved, effectively preventing overfitting on limited localized visibility datasets, while allowing higher-level networks to fine-tune for visibility-related semantic features; the meteorological branch uses a simple multilayer perceptron, sufficient to handle the nonlinear relationships between filtered low-dimensional meteorological factors. This invention abandons complex custom networks, reducing the overall complexity and computational cost of the model while ensuring effective utilization of multimodal information.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a visibility monitoring method, system, terminal, and medium based on image deep learning. Background Technology

[0002] Visibility is a key meteorological indicator affecting traffic safety, aviation operations, and environmental monitoring. Timely and accurate automated monitoring of low visibility conditions is of great significance. Traditional visibility monitoring mainly relies on manual visual inspection or specialized instruments, which has limitations such as strong subjectivity, high cost, and sparse deployment. With the development of computer vision and deep learning technologies, automated visibility estimation methods based on images and meteorological data have become a research hotspot. Among them, multimodal methods that integrate image and meteorological information, compared with single-data source methods, can comprehensively utilize visual features and physical causes, and theoretically can obtain more robust and accurate monitoring results.

[0003] Several deep learning-based multimodal visibility estimation schemes have emerged in the existing technology. For example, Chinese patent document CN113723199A discloses an airport low visibility detection method, which trains a meteorological factor prediction model and a cross-modal prediction model separately, and then fuses the outputs of the two models through an additional weight model to output the final visibility prediction value. Although this method improves reliability through multi-model fusion, its process involves the training and concatenation of three independent models, resulting in a complex structure, high computational and maintenance costs, and its final output is a specific visibility value, which is inefficient in application scenarios requiring rapid level discrimination. Another example is Chinese patent document CN118262204A, which proposes a highway visibility prediction method based on multimodal and multi-scale feature fusion. It constructs a complex network including Bi-LSTM (Bidirectional Long Short-Term Memory), multi-head attention, and CNN (Convolutional Neural Network) to extract temporal features, and uses algorithms such as SIFT (Scale Invariant Feature Transform) and spatial pyramid pooling to extract image features, and then performs multi-scale fusion. Although the scheme considers features at different time scales, its feature extraction networks, such as Att-BiCNet (a bidirectional long short-term memory network based on attention mechanism combined with convolutional neural networks) and SIFT-SPPNet (a spatial pyramid pooling network based on scale-invariant feature transformation), are extremely complex. This results in a large number of model parameters, high computational costs for training and inference, and reliance on non-standard network components. This makes model optimization, acceleration, and deployment in real-world engineering environments extremely challenging, and makes it difficult to meet the localized monitoring requirements that demand real-time performance, lightweight design, and ease of deployment.

[0004] Therefore, researching and designing a visibility monitoring method, system, terminal, and medium based on image deep learning that can overcome the above-mentioned defects is an urgent problem to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a visibility monitoring method, system, terminal, and medium based on image deep learning. It utilizes the powerful general visual feature extraction capabilities of a ResNet network pre-trained on a massive dataset as a strong baseline for processing visibility images. By freezing the parameters of its shallow and mid-layers, the universal representation of basic features such as edges and textures is preserved, effectively preventing overfitting on limited localized visibility datasets. Simultaneously, it allows higher-level networks to fine-tune for visibility-related semantic features. The meteorological branch employs a simple multilayer perceptron, sufficient to handle the nonlinear relationships between filtered low-dimensional meteorological factors. This invention abandons complex custom networks, reducing the overall complexity and computational cost of the model while ensuring effective utilization of multimodal information, thus laying the foundation for lightweight and real-time deployment of visibility monitoring models.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: Firstly, a visibility monitoring method based on image deep learning is provided, including the following steps: Acquire visibility image data of the target monitoring area and meteorological element data synchronized with the image data; The visibility image data is input into the visual feature extraction branch of a pre-trained visibility estimation model to obtain an image feature vector; wherein, the visual feature extraction branch uses a pre-trained residual convolutional neural network as the backbone network, and the network parameters of at least the first N processing stages used to extract primary and intermediate features in the backbone network are fixed during the model training process. The meteorological element data is input into the meteorological feature extraction branch based on multilayer perceptron in the pre-trained visibility estimation model to obtain the meteorological feature vector; The image feature vector and the meteorological feature vector are fused to obtain the fused feature; Based on the fused features, the current visibility level is determined by a classifier.

[0007] Furthermore, the residual convolutional neural network is a ResNet-101 network, and the first N processing stages include an initial convolutional layer, a max pooling layer, and a first residual block stage.

[0008] Furthermore, before inputting the visibility image data into the visual feature extraction branch of the pre-trained visibility estimation model, the ResNet-101 network is modified, including: Remove the original classification layer at the end of the ResNet-101 network; An adaptive pooling layer is added after the feature extraction end of the ResNet-101 network to convert the extracted feature map into a fixed-dimensional image feature vector.

[0009] Furthermore, the process of acquiring the meteorological element data includes: Obtain raw meteorological element data; The correlation between various meteorological elements and measured visibility values ​​was calculated based on historical data. Based on the correlation analysis results, key meteorological elements are selected from the original meteorological element data to form an input subset; Principal component analysis is performed on the data in the input subset to obtain the meteorological element data after dimensionality reduction.

[0010] Furthermore, the key meteorological elements include relative humidity, local air pressure, standard value of precipitation per minute, 2-minute average wind speed, maximum instantaneous wind speed within minutes, and 10-minute average wind speed.

[0011] Furthermore, the multilayer perceptron in the meteorological feature extraction branch includes a linear layer, a batch normalization layer, a ReLU activation function layer, and a Dropout layer arranged sequentially.

[0012] Furthermore, the classifier includes a first fully connected layer, a batch normalization layer, a ReLU activation function layer, a Dropout layer, and a second fully connected layer connected in sequence. The first fully connected layer is used to map the fused features to the intermediate dimension. The number of neurons in the second fully connected layer is the same as the number of categories in the visibility level, and its output is converted into the probability distribution of each category by the Softmax function.

[0013] Secondly, a visibility monitoring system based on image deep learning is provided, including: The data acquisition module is used to acquire visibility image data of the target monitoring area and meteorological element data synchronized with the image data; The image processing module is used to input the visibility image data into the visual feature extraction branch of the pre-trained visibility estimation model to obtain image feature vectors; wherein, the visual feature extraction branch uses a pre-trained residual convolutional neural network as the backbone network, and the network parameters of at least the first N processing stages used to extract primary and intermediate features in the backbone network are fixed during the model training process. The meteorological processing module is used to input the meteorological element data into the meteorological feature extraction branch based on multilayer perceptron in the pre-trained visibility estimation model to obtain meteorological feature vectors. The feature fusion module is used to fuse the image feature vector and the meteorological feature vector to obtain fused features; The classification decision module is used to determine the current visibility level based on the fused features using a classifier.

[0014] Thirdly, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visibility monitoring method based on image deep learning as described in any one of the first aspects.

[0015] Fourthly, a computer-readable medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the visibility monitoring method based on image deep learning as described in any one of the first aspects.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The visibility monitoring method based on deep learning for images provided by this invention utilizes the powerful general visual feature extraction capability of the ResNet network pre-trained on a large-scale dataset as a strong baseline for processing visibility images. By freezing its shallow and mid-layer parameters, the universal representation of basic features such as edges and textures of these layers is preserved, effectively preventing overfitting on limited localized visibility datasets. At the same time, it allows high-level networks to fine-tune for visibility-related semantic features. The meteorological branch adopts a simple multilayer perceptron, which is sufficient to handle the nonlinear relationships between the selected low-dimensional meteorological factors. This invention abandons complex custom networks and reduces the overall complexity and computational cost of the model while ensuring the effective utilization of multimodal information, laying the foundation for the lightweight and real-time deployment of the visibility monitoring model.

[0017] 2. This invention freezes the shallow layers, including conv1, maxpool, and conv2_x. During training on local visibility data, the network parameters responsible for extracting the most basic and general visual patterns remain unchanged. This allows the model to stably inherit the powerful prior knowledge from ImageNet pre-trained weights, requiring only fine-tuning of the deeper network parts responsible for extracting higher-level semantic features more relevant to visibility. This strategy greatly enhances adaptability to small-scale local datasets, effectively avoiding the risk of model performance degradation or overfitting due to limited data. Thus, even under resource-constrained conditions, a high-precision visibility level classification model adapted to local climate characteristics can still be obtained.

[0018] 3. In meteorological data processing, this invention can eliminate redundant meteorological factors with weak or no linear relationship with visibility through correlation coefficient analysis, focusing on core influencing elements. Principal component analysis can further eliminate multicollinearity that may exist among the screened elements, transforming the original data into a set of uncorrelated principal components that can retain the variance of the original data to the greatest extent. This provides the model with a set of purified and compressed meteorological state fingerprint features, reducing noise and redundant information input to the model from the source, reducing the learning difficulty of meteorological feature extraction branches, and improving the efficiency and robustness of the model in modeling the complex relationship between meteorological conditions and visibility. This makes the final fused features more discriminative and the classification results more reliable. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall architecture diagram of the visibility estimation model in Embodiment 1 of the present invention; Figure 2 This is a probability density distribution diagram of the 2-minute average wind speed in Embodiment 1 of the present invention; Figure 3 This is the probability density distribution diagram of the 2-minute average wind direction in Embodiment 1 of the present invention; Figure 4 This is a probability density distribution diagram of the 10-minute average wind speed in Embodiment 1 of the present invention; Figure 5 This is the probability density distribution diagram of the 10-minute average wind direction in Embodiment 1 of the present invention; Figure 6 This is a probability density distribution diagram of the air pressure at this station in Embodiment 1 of the present invention; Figure 7 This is a probability density distribution diagram of relative humidity in Embodiment 1 of the present invention; Figure 8 This is a probability density distribution diagram of the maximum instantaneous wind speed within a minute in Embodiment 1 of the present invention; Figure 9 This is a probability density distribution map of the maximum instantaneous wind direction within a minute in Embodiment 1 of the present invention; Figure 10 This is a probability density distribution diagram of the standard temperature value in Embodiment 1 of the present invention; Figure 11 This is a probability density distribution diagram of the standard value of minute precipitation in Embodiment 1 of the present invention; Figure 12 This is a probability density distribution diagram of the standard value of hourly cumulative precipitation in Embodiment 1 of the present invention; Figure 13 This is a network structure diagram of the visual feature extraction branch in Embodiment 1 of the present invention; Figure 14 This is a network structure diagram of the meteorological feature extraction branch in Embodiment 1 of the present invention; Figure 15 This is a network structure diagram of feature fusion and classification in Embodiment 1 of the present invention; Figure 16 This is a diagram showing the result of the normalized confusion matrix in Embodiment 1 of the present invention; Figure 17 This is a system block diagram in Embodiment 2 of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0021] Example 1: A visibility monitoring method based on image deep learning, such as Figure 1 As shown, it includes the following steps: S1: Acquire visibility image data of the target monitoring area and meteorological element data synchronized with the image data; S2: Input the visibility image data into the visual feature extraction branch of the pre-trained visibility estimation model to obtain the image feature vector; wherein, the visual feature extraction branch uses a pre-trained residual convolutional neural network as the backbone network, and the network parameters of at least the first N processing stages used to extract primary and intermediate features in the backbone network are fixed during the model training process. S3: Input meteorological element data into the meteorological feature extraction branch based on multilayer perceptron in the pre-trained visibility estimation model to obtain meteorological feature vectors; S4: Fuse the image feature vector and the meteorological feature vector to obtain the fused feature; S5: Based on the fused features, the current visibility level is determined by a classifier.

[0022] In step S1, in this embodiment, the target monitoring area is selected as a representative measurement field in a certain district. To achieve visibility monitoring, it is necessary to simultaneously acquire both image and meteorological data. Image data comes from high-definition cameras deployed at the measurement field, which automatically capture and store visibility monitoring images every 5 minutes. Meteorological data is collected synchronously from minute-level meteorological observation equipment at the same measurement point. The recorded raw meteorological element data includes, but is not limited to, relative humidity, local air pressure, standard value of minute precipitation, standard value of hourly cumulative precipitation, standard value of temperature, 2-minute average wind speed, maximum instantaneous wind speed within minutes, 10-minute average wind speed, wind direction of maximum instantaneous wind speed within minutes, 2-minute average wind direction, and 10-minute average wind direction. After data acquisition, each visibility monitoring image is aligned with all meteorological element observation records corresponding to the shooting time using precise timestamps to ensure strict synchronization of multimodal data in the time dimension. For individual data gaps or obvious outliers generated during transmission or storage, corresponding cleaning and imputation processing is performed to ensure the integrity of the dataset.

[0023] After obtaining the aligned synchronous dataset, in order to build an efficient visibility estimation model, the original meteorological elements need to be screened and dimensionality reduced to eliminate redundancy and extract core influencing factors. First, based on long-term historical data, the Pearson correlation coefficient between each meteorological element and the visibility value measured by the visibility meter is calculated to quantify the strength of their linear correlation.

[0024] like Figures 2-12 As shown, the calculation results indicate that the correlation coefficient between relative humidity and visibility is -0.766, showing a significant negative correlation; the correlation coefficients between 2-minute average wind speed, maximum instantaneous wind speed within minutes, and 10-minute average wind speed and temperature are 0.46, 0.504, 0.507, and 0.402, respectively, showing a significant positive correlation; while the correlation between station air pressure, minute precipitation, and wind direction is relatively weak.

[0025] To more accurately select modeling factors, feature importance analysis was further conducted based on correlation analysis. Random forest and other algorithms were used to evaluate the contribution of each meteorological element to visibility changes and quantify their influence weights. The analysis results showed that relative humidity had the highest feature importance score (3.43), while the scores for local air pressure, standard value of minute precipitation, 2-minute average wind speed, maximum instantaneous wind speed within minutes, and 10-minute average wind speed were also significantly high. The calculation results are shown in Table 1. Combining the correlation coefficient and feature importance scores, relative humidity, local air pressure, standard value of minute precipitation, 2-minute average wind speed, maximum instantaneous wind speed within minutes, and 10-minute average wind speed were initially selected as the six key meteorological elements, forming the initial input subset. These six elements cover the core physical processes affecting visibility, including water vapor conditions (humidity, precipitation), dynamic conditions (wind speed), and thermal conditions (indirectly reflected through air pressure).

[0026] Table 1 Calculation results of various meteorological factors

[0027] To address the potential multicollinearity issue among the initially selected key meteorological elements, such as the high correlation between wind speed indices at different time scales, and to further reduce data dimensionality and improve model efficiency, principal component analysis (PCA) was performed on the six-dimensional key meteorological element data. First, the observed values ​​of the six elements were standardized to eliminate the influence of dimensions. Then, the covariance matrix of the standardized data was calculated, and the eigenvalues ​​and eigenvectors of this matrix were solved. The magnitude of the eigenvalues ​​reflects the amount of original data information contained in the corresponding principal component. Principal components were selected in descending order of eigenvalues, ensuring that the cumulative variance contribution rate of the selected principal components reached or exceeded 85%, thereby ensuring that the dimensionality-reduced data retained most of the core variation information of the original key meteorological elements. Finally, the original six-dimensional key meteorological element data was projected onto a few selected principal components, resulting in a set of new, uncorrelated, lower-dimensional eigenvectors. This set of eigenvectors obtained after dimensionality reduction through PCA serves as the final input meteorological element data to the subsequent meteorological feature extraction branch, thus achieving the goal of effectively reducing the model input dimensionality and improving computational efficiency while retaining the main meteorological information.

[0028] In step S2, as Figure 13 As shown, the visual feature extraction branch uses a residual convolutional neural network (ResNet-101, Residual Network with 101 layers) pre-trained on the ImageNet (a large image recognition database) large-scale image dataset as the backbone network. The pre-trained model was chosen to leverage its learned general visual feature extraction capabilities, such as basic and intermediate semantic features like edges, textures, and shapes. This provides a strong prior knowledge base for effectively identifying visual patterns related to atmospheric transparency from visibility monitoring images and accelerates the model's convergence on this specific task. Visual patterns include haze concentration, the degree of blurring in distant scenes, and the contrast between sky and ground features.

[0029] To achieve efficient model adaptation to local visibility monitoring tasks and prevent overfitting on small datasets, the parameters of the front-end network layers in the ResNet-101 backbone responsible for extracting primary and intermediate image features are fixed during model training—a process known as "freezing." Specifically, the frozen network processing stages include: the initial 7x7 convolutional layer (conv1, one-dimensional convolution), the subsequent 3x3 max-pooling layer, and the first residual block stage consisting of three stacked residual blocks (conv2_x). Freezing these layers means that during subsequent training on the visibility dataset, only the parameters of deeper network layers such as conv3_x, conv4_x, and conv5_x, as well as newly added classification modules, are updated via backpropagation, while the weights of the frozen layers remain unchanged. This strategy ensures that the general feature detectors at the lower layers of the network are preserved, while allowing the higher layers to fine-tune feature representations for the specific task of visibility estimation.

[0030] When constructing the visual feature extraction branch, the pre-trained original ResNet-101 network structure needs to be adaptively modified to adapt its output to subsequent multimodal feature fusion. First, the original classification layer at the end of the ResNet-101 network is removed. This layer was originally a global average pooling layer followed by a 1000-dimensional fully connected layer, used for 1000 object classification classes in ImageNet. After removal, the remaining convolutional and pooling layers will serve as pure feature extractors. Next, at the end of the ResNet-101 network's feature extraction, after the output of the last residual block stage conv5_x, an adaptive average pooling layer is added. The function of this layer is to pool the feature map output by the previous convolutional layer to a fixed 1x1 spatial size, regardless of the specific spatial dimensions, while retaining all feature channels. This operation transforms the two-dimensional feature map into a one-dimensional feature vector. Specifically, for each feature channel d in the input feature map, the adaptive average pooling operation can be described as follows: Performing this operation on all 2048 channels yields a 2048-dimensional image feature vector. This vector encapsulates the global visual semantic information extracted from the input visibility monitoring image via a deep convolutional network, providing crucial input for subsequent fusion and classification.

[0031] In step S3, as Figure 14As shown, the meteorological feature extraction branch is a multilayer perceptron specifically designed for processing structured numerical data. The input to this branch is the dimensionality-reduced meteorological element data obtained through the preprocessing step S1. Before input, this data needs to be standardized to eliminate the influence of differences in dimensions and numerical ranges between different meteorological elements, ensuring the stability and convergence speed of model training. Standardization typically employs the Z-Score method (a standardization method that sets the data mean to 0 and the standard deviation to 1), performed separately for each meteorological feature dimension.

[0032] Specifically, a multilayer perceptron consists of a series of sequentially connected neural network layers, designed to extract high-level, nonlinear feature representations from low-dimensional meteorological data. Its first layer is a linear layer that performs a linear transformation of the feature vectors.

[0033] Following the linear layer, a batch normalization layer is connected. This layer standardizes the output of each feature dimension in the current batch of data, making its mean 0 and variance 1. Its calculation typically involves scaling and translation operations. Batch normalization effectively alleviates the internal covariate shift problem during training, accelerates network convergence, and has a certain regularization effect.

[0034] Subsequently, the network introduces nonlinear transformation capabilities through a ReLU (Rectified Linear Unit) activation function layer. The ReLU function acts on each output component of the batch normalization layer, setting all negative input values ​​to zero while leaving positive input values ​​unchanged. This sparse activation characteristic enables the model to learn more complex nonlinear patterns, which is crucial for characterizing the complex, nonlinear interactions between meteorological elements and visibility. Furthermore, its computational simplicity helps alleviate the gradient vanishing problem in deep networks.

[0035] Following the ReLU activation function layer, the network introduces a Dropout layer as a regularization mechanism to prevent overfitting. During forward propagation, the Dropout layer randomly discards some neuron activation values ​​from the ReLU activation function layer with a preset probability p, temporarily setting them to 0. Each neuron being discarded is an independent event. This mechanism forces the network to work collaboratively, rather than relying on a few specific neurons, thus learning more robust features and improving the model's generalization ability. In the prediction phase, all neurons participate in the computation, but their output values ​​are multiplied by a retention probability of 1. p is used for scaling.

[0036] After layer-by-layer processing by the multilayer perceptron, the original meteorological element data is transformed into a more expressive, fixed-dimensional feature vector. In a specific implementation of this embodiment, the meteorological feature extraction branch ultimately outputs a 256-dimensional meteorological feature vector. This vector encodes abstracted and condensed meteorological state information and will be input together with the image feature vector output by the visual feature extraction branch into subsequent feature fusion.

[0037] In step S4, as Figure 15 As shown, the system receives image feature vectors from the visual feature extraction branch and meteorological feature vectors from the meteorological feature extraction branch, and integrates them to form a fused feature containing joint visual and meteorological information, providing a comprehensive data foundation for the final visibility level classification. The feature fusion strategy employed is vector concatenation, a simple, efficient, and widely used multimodal feature fusion method. Its advantage lies in its ability to completely preserve the original feature information from two different modalities, avoiding information loss that may occur during early fusion, and providing sufficient discriminative basis for subsequent classifiers.

[0038] Specifically, the fusion operation is performed along the feature dimension. Let the image feature vector output by the visual feature extraction branch be denoted as... Its dimension is 2048; the meteorological feature vector output by the meteorological feature extraction branch is denoted as Its dimension is 256. The feature fusion module performs a concatenation operation, combining... Connected to The tail of the feature vector is then used to generate a new, longer feature vector. This process can be expressed by the formula: After concatenation, the feature vectors are merged. The fusion feature vector has 2304 dimensions. This 2304-dimensional fusion feature vector simultaneously encodes the deep visual semantics of the input visibility monitoring image and the high-level abstract representation of the synchronous meteorological conditions, achieving effective alignment and complementarity of visual and meteorological information at the feature level, thus creating conditions for the model to make comprehensive judgments. This fusion feature is then sent to the classification decision for final processing.

[0039] In step S5, as Figure 15 As shown, the system receives the fused feature vector from the feature fusion output and processes it through a multi-layered classifier, ultimately outputting the visibility level determination result for the current monitoring time. The classifier is designed to learn complex decision boundaries from the fused features to distinguish different visibility levels.

[0040] The first layer of the classifier is a fully connected layer, whose main function is to perform dimensionality reduction and nonlinear feature transformation on the high-dimensional fused features. This layer receives a 2304-dimensional fused feature vector as input and maps it to an intermediate feature space with a more moderate dimension and greater discriminativeness. In this embodiment, it is mapped to 512 dimensions.

[0041] Following the first fully connected layer is a batch normalization layer (BatchNorm), which standardizes each feature dimension in the current batch of data, adjusting it to a distribution with a mean of 0 and a variance of 1. Learnable scaling and translation parameters are introduced to stabilize the distribution of the intermediate layer inputs, accelerate training convergence, and improve the model's generalization ability.

[0042] Next, the network introduces nonlinearity through a ReLU activation function layer. This function performs a negative zeroing operation on each feature value after batch normalization, activating all negative values ​​to zero while keeping positive values ​​unchanged. This allows the model to fit the complex nonlinear relationship between fused features and visibility levels and helps alleviate the gradient vanishing problem.

[0043] Following the ReLU activation function layer, a Dropout layer is set as a regularizer. During forward propagation training, this layer randomly sets the activation values ​​of some neurons in the output of the previous layer to 0 with a preset probability. This mechanism reduces overfitting to the training data by preventing complex co-adaptation between neurons, forcing the network to learn more robust feature representations, thereby improving the model's visibility level classification performance on unseen data.

[0044] The final layer of the classifier is the second fully connected layer, whose function is to map the features processed by the previous layers to the final visibility category space. The number of neurons in this layer is strictly set to the total number of visibility level categories, and the visibility level classification criteria are shown in Table 2.

[0045] Table 2 Visibility Level Classification Standards

[0046] In order to transform the original score vector output by the second fully connected layer The visibility is transformed into an intuitive probability distribution to determine the final visibility level. Each element of the original score vector is exponentially normalized using the Softmax function, calculated as follows: ; in, The model predicts that the current sample belongs to the first... The function calculates the probability of each visibility level. It ensures that the sum of all output probabilities is 1. Finally, the category index with the highest probability value is determined as the current visibility level, thus completing a fully automated monitoring process from multimodal data input to specific visibility condition determination.

[0047] In some optional examples, during the model training phase, to address the issue of degraded minority class recognition performance caused by a severe imbalance in the number of samples across different visibility levels in the original dataset, this embodiment employs a stratified sampling strategy to construct a balanced training dataset. Specifically, firstly, based on the visibility level classification criteria, all aligned multimodal data samples are divided into five subsets. Subsequently, a predetermined number of samples are independently and randomly drawn from each level's subset. The predetermined number can be set as the minimum number of samples across all levels, or a target number can be set for each level based on actual needs, such as oversampling levels with fewer samples. Finally, the samples drawn from all levels are merged to form a training dataset with a balanced class distribution. Using this balanced dataset for joint training of the visual feature extraction branch, the meteorological feature extraction branch, and the classifier can effectively prevent the model from biased towards learning features of classes with a large number of samples, thereby significantly improving the model's ability to identify and warn of all visibility levels, especially low visibility weather.

[0048] After data cleaning, the original dataset contained 23,395 data points, of which 21,702 data points had a visibility of more than 2,000 kilometers, accounting for more than 90%, as shown in Table 3.

[0049] Table 3. Percentage of different visibility levels in the total dataset

[0050] Data from different visibility levels were sampled, with a total of 2006 samples. The sampled dataset was then split into training and testing sets in an 8:2 ratio. The sampling results are shown in Table 4.

[0051] Table 4. Distribution of the dataset after sampling

[0052] In some optional examples, to ensure the efficient and stable operation of the trained visibility estimation model in a production environment, this embodiment introduces the TorchScript toolchain for model conversion and optimization during the model deployment phase. The specific process is as follows: First, a complete model trained and converged on a balanced dataset is loaded, including the modified ResNet-101 visual branch in S2, the multilayer perceptron meteorological branch in S3, the feature fusion logic in S4, and the classifier in S5. Then, using the `torch.jit.trace` function provided by the PyTorch framework, combined with a representative example input, including a visibility image tensor of the expected size and a meteorological element data tensor of the corresponding dimension, the forward propagation process of the model is traced. This tracing process records all operations performed as the data flows through each computation node of the model, and generates a static computation graph representation independent of the original Python source code. TorchScript serializes this computation graph into a model file in .pt or .pth format. This serialized model file contains the model structure, parameters, and optimized computational logic, but no longer depends on the Python environment used during training. During deployment, this file can be directly loaded by inference services written in languages ​​such as C++ and Java, or called through PyTorch's C++ frontend LibTorch. This transformation not only enables cross-platform deployment of the model and eliminates the dependency on the Python interpreter, but also significantly improves the inference speed of the model on CPUs or GPUs and reduces the service's memory overhead because the computation graph is pre-optimized at load time. This meets the needs of real-world business applications for high-concurrency, low-latency, real-time visibility monitoring.

[0053] In some optional examples, to transform the aforementioned visibility monitoring capabilities into usable business functions, this embodiment describes an automated monitoring pipeline that integrates the trained and transformed model into an existing meteorological or traffic business system. This pipeline first acquires the latest visibility images and raw meteorological data from cameras and meteorological sensors deployed in the target monitoring area at a predetermined frequency. Next, the raw data is automatically timestamped and the same preprocessing procedure as in embodiment S1 is invoked. The preprocessed data is then fed into an inference engine loaded with the aforementioned TorchScript serialization model. The inference engine executes procedures S2 to S5, quickly outputting the visibility level and its probability at the current moment. Finally, the determination result, along with time, location, and other information, is automatically pushed to the corresponding display interface or database of the next-level business platform via an application programming interface (API), and alarm information can be triggered based on preset thresholds. This integration automates the entire process from data acquisition, automatic processing, intelligent analysis to result reporting, enabling visibility monitoring methods to be seamlessly embedded into existing technology architectures and providing near real-time, high-precision visibility information services 24 / 7.

[0054] This invention employs stratified sampling to construct a balanced dataset during model training and utilizes TorchScript for model transformation during deployment. Stratified sampling ensures balanced representativeness of each visibility level in the training set. Its technical logic forces the model to equally focus on all categories during learning, especially the low-visibility category with few samples. This effectively alleviates the problem of poor model performance in minority class recognition caused by imbalanced original data distribution, significantly improving the model's early warning capability for low-visibility hazardous weather such as fog and haze. The TorchScript tool (a tool provided by the PyTorch framework that converts Python code into an intermediate representation independent of the Python environment and callable and serializable by high-performance languages ​​such as C++) is used to serialize the trained model. Its principle is to convert the dynamic PyTorch model into a static computation graph, generating independent model files. This allows the model to be deployed directly by high-performance languages ​​such as C++, without being trained in the Python environment. This enables out-of-the-box deployment of the model, significantly improves inference speed, reduces server-side resource consumption, and solves the engineering challenges of deploying complex deep learning models in production environments, which are cumbersome, have complex dependencies, and suffer from high latency. This makes the visibility monitoring solution of this invention truly capable of large-scale, high-concurrency, real-time business applications.

[0055] Experimental verification: Since visibility estimation is a multi-classification task, in order to objectively and comprehensively evaluate the performance of this model, three evaluation metrics widely used in classification tasks are adopted: accuracy, recall and F1 score. These metrics are all calculated based on the confusion matrix of the prediction results of the test samples.

[0056] For the five-class classification task in this model, the confusion matrix is ​​a 5x5 matrix where rows represent the true class of the sample and columns represent the predicted class of the model. Each element in the matrix... This represents the number of samples whose true class is i but are predicted as class j. The elements on the diagonal represent samples correctly classified by the model. Based on the confusion matrix, metrics for each class can be defined: The macro-averaging strategy treats all five categories equally, calculating the metric for each category independently before taking the arithmetic mean. Therefore, it reflects the model's performance equally across all categories, preventing a single category's larger sample size from dominating the overall evaluation. Finally, the overall accuracy, the proportion of correctly classified samples out of all categories, is calculated to measure the model's overall performance.

[0057] As shown in Table 5, the model achieved a macro F1 score of 90.91% across the five categories, which fully demonstrates that the model can effectively integrate image features and structured data features, and achieve balanced and excellent discrimination ability across all categories.

[0058] Meanwhile, the model's macro-accuracy and macro-recall reached 91.30% and 91.10% respectively, both of which are close and at a high level. This indicates that while maintaining high accuracy, the model also possesses excellent recall capabilities, effectively avoiding the problem of missed detections caused by pursuing high accuracy. In addition, the model's overall accuracy was 92.29%, which intuitively reflects the model's high classification accuracy on the overall sample.

[0059] Table 5. Experimental Results of Visibility Estimation Model

[0060] To analyze the model's classification behavior and error patterns in greater depth, a normalized confusion matrix was plotted, such as... Figure 16 As shown in the diagram, the matrix is ​​normalized row by row. The values ​​on the diagonal visually represent the recall rate for each category, while the off-diagonal elements reveal the main confusion between categories.

[0061] Analysis of the confusion matrix revealed that: (1) the main diagonal of the matrix was dark in color and the values ​​were generally higher than 0.90, indicating that the model maintained a high recall rate for all categories, which is consistent with the aforementioned macro recall rate (91.10%). (2) The main classification errors of the model were concentrated between adjacent visibility levels. For example, about 12% of level 4 visibility samples were misclassified as level 3, while 10% of level 5 visibility samples were misclassified as level 4. This confusion pattern is consistent with the characteristics of human visual perception, because adjacent visibility levels are inherently highly similar in appearance. (3) It is worth noting that the model rarely misclassifies across levels, such as directly misclassifying level 1 visibility as level 3 or above, or misclassifying level 5 visibility as level 3 or below. This proves that the features learned by the model have strong discriminative power and can accurately grasp the essential differences between different visibility levels.

[0062] In summary, the combined indicators and confusion matrix analysis show that the model not only has superior overall performance, but its error patterns also conform to objective laws, demonstrating good robustness and practicality.

[0063] To verify the effectiveness of this model, it was compared with three representative state-of-the-art vision models. All baseline models were trained on the same training set, using the same preprocessing pipeline, and evaluated on the same test set to ensure fairness in the comparison. The results are shown in Table 6.

[0064] Table 6 Macro-average indices for different models

[0065] The comparison results in the table above show that: (1) This model outperforms all baseline models in all evaluation metrics. Specifically, in the core metric macro F1 score, this model achieves 91.10%, which is 1.23, 14.68, and 5.21 percentage points higher than ConvNeXt, ResNet-101, and Vision Transformer, respectively. This result strongly demonstrates that introducing and fusing structured data through the MLP (Multilayer Perceptron) branch can provide complementary information for visibility estimation that the image itself does not possess, thereby significantly improving the overall discriminative performance of the model. (2) Among the pure visual baseline models, the performance ranking is ConvNeXt > Vision Transformer > ResNet-101. As a modern variant of convolutional neural networks, ConvNeXt, with its ingenious architectural design, performs best in the dense visual perception task of visibility estimation, becoming the strongest pure visual baseline. Vision Transformer demonstrated powerful modeling capabilities, but its performance was slightly inferior to ConvNeXt. This may be because, given the limited dataset size, the Transformer architecture is more dependent on inductive bias. ResNet-101, as a representative of traditional CNNs, showed a significant performance gap compared to advanced architectures, highlighting the importance of model architecture innovation in improving performance. (3) This model uses ResNet as the backbone network, but its performance far exceeds that of ResNet-101. This not only reflects the effectiveness of the model design, but more importantly, it clearly demonstrates the success of the multimodal fusion strategy. The structured data processed by the MLP branch effectively compensates for the shortcomings of the pure vision model. In addition, this model also achieved a stable improvement over the highest-performing pure vision baseline (ConvNeXt), indicating that multimodal information is not simply redundant, but provides truly valuable new discriminative dimensions.

[0066] Furthermore, this invention demonstrates significant advantages in image feature extraction networks, inference speed, deployment flexibility, and adaptability to various application scenarios. For the backbone network selection, this invention employs ResNet101, which, compared to existing technologies relying on VisionTransformer, features a moderate number of parameters and fast inference speed, reducing hardware computing power requirements and making it more suitable for edge device deployment. In terms of applicability, this invention breaks the limitation of traditional methods that only address low visibility or extreme fog / haze scenarios within 500 meters, achieving full coverage from good to severe conditions. This comprehensive adaptability makes it suitable not only for specialized fields such as airports but also for diverse scenarios such as urban transportation, event support, and tourism, possessing stronger cross-scenario generalization capabilities.

[0067] In summary, the experimental results fully validate the rationality of the proposed model's design and its advanced performance. Compared to advanced models that rely on a single visual modality, this model achieves more accurate and robust classification performance by effectively fusing information from multiple sources.

[0068] Example 2: A visibility monitoring system based on image deep learning. This system is used to implement the visibility monitoring method based on image deep learning described in Example 1, such as... Figure 17 As shown, it includes a data acquisition module, an image processing module, a meteorological processing module, a feature fusion module, and a classification decision module.

[0069] The system comprises the following modules: a data acquisition module for acquiring visibility image data of the target monitoring area and meteorological element data synchronized with the image data; an image processing module for inputting the visibility image data into the visual feature extraction branch of a pre-trained visibility estimation model to obtain image feature vectors; wherein the visual feature extraction branch uses a pre-trained residual convolutional neural network as the backbone network, and the network parameters of at least the first N processing stages used to extract primary and intermediate features in the backbone network are fixed during model training; a meteorological processing module for inputting meteorological element data into the meteorological feature extraction branch based on a multilayer perceptron in the pre-trained visibility estimation model to obtain meteorological feature vectors; a feature fusion module for fusing the image feature vectors and meteorological feature vectors to obtain fused features; and a classification decision module for determining the current visibility level based on the fused features using a classifier.

[0070] Example 3: A computer terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the visibility monitoring method based on image deep learning as described in Example 1.

[0071] Example 4: A computer-readable medium having a computer program stored thereon, which, when executed by a processor, enables the visibility monitoring method based on image deep learning as described in Example 1.

[0072] Working principle: This invention utilizes the powerful general visual feature extraction capabilities of a ResNet network pre-trained on a massive dataset as a strong baseline for processing visibility images. By freezing the parameters of its shallow and mid-layers, the universal representation of basic features such as edges and textures is preserved, effectively preventing overfitting on limited localized visibility datasets. Simultaneously, it allows higher-level networks to fine-tune for visibility-related semantic features. The meteorological branch employs a simple multilayer perceptron, sufficient to handle the nonlinear relationships between selected low-dimensional meteorological factors. This invention abandons complex custom networks, reducing the overall complexity and computational cost of the model while ensuring effective utilization of multimodal information. This lays the foundation for lightweight and real-time deployment of visibility monitoring models.

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image depth learning-based visibility monitoring method, characterized in that, Includes the following steps: Acquire visibility image data of the target monitoring area and meteorological element data synchronized with the image data; The visibility image data is input into the visual feature extraction branch of a pre-trained visibility estimation model to obtain an image feature vector; wherein, the visual feature extraction branch uses a pre-trained residual convolutional neural network as the backbone network, and the network parameters of at least the first N processing stages used to extract primary and intermediate features in the backbone network are fixed during the model training process. The meteorological element data is input into the meteorological feature extraction branch based on multilayer perceptron in the pre-trained visibility estimation model to obtain the meteorological feature vector; The image feature vector and the meteorological feature vector are fused to obtain the fused feature; Based on the fused features, the current visibility level is determined by a classifier.

2. The image depth learning-based visibility monitoring method according to claim 1, characterized in that, The residual convolutional neural network is a ResNet-101 network, and the first N processing stages include an initial convolutional layer, a max pooling layer, and a first residual block stage.

3. The image depth learning-based visibility monitoring method according to claim 2, characterized in that, Before inputting the visibility image data into the visual feature extraction branch of the pre-trained visibility estimation model, the ResNet-101 network is modified, including: Remove the original classification layer at the end of the ResNet-101 network; An adaptive pooling layer is added after the feature extraction end of the ResNet-101 network to convert the extracted feature map into a fixed-dimensional image feature vector.

4. The image depth learning-based visibility monitoring method according to claim 1, characterized in that, The process of acquiring the meteorological element data includes: Obtain raw meteorological element data; The correlation between various meteorological elements and measured visibility values ​​was calculated based on historical data. Based on the correlation analysis results, key meteorological elements are selected from the original meteorological element data to form an input subset; Principal component analysis is performed on the data in the input subset to obtain the meteorological element data after dimensionality reduction.

5. The image depth learning-based visibility monitoring method according to claim 4, characterized in that, The key meteorological elements include relative humidity, local air pressure, standard value of precipitation per minute, 2-minute average wind speed, maximum instantaneous wind speed within minutes, and 10-minute average wind speed.

6. The image depth learning-based visibility monitoring method according to claim 1, characterized in that, The multilayer perceptron in the meteorological feature extraction branch includes a linear layer, a batch normalization layer, a ReLU activation function layer, and a Dropout layer arranged sequentially.

7. The image depth learning-based visibility monitoring method according to claim 1, characterized in that, The classifier comprises a first fully connected layer, a batch normalization layer, a ReLU activation function layer, a Dropout layer, and a second fully connected layer connected in sequence. The first fully connected layer is used to map the fused features to the intermediate dimension. The number of neurons in the second fully connected layer is the same as the number of categories in the visibility level, and its output is converted into the probability distribution of each category by the Softmax function.

8. The visibility monitoring system based on image deep learning, characterized in that, include: The data acquisition module is used to acquire visibility image data of the target monitoring area and meteorological element data synchronized with the image data; The image processing module is used to input the visibility image data into the visual feature extraction branch of the pre-trained visibility estimation model to obtain image feature vectors; wherein, the visual feature extraction branch uses a pre-trained residual convolutional neural network as the backbone network, and the network parameters of at least the first N processing stages used to extract primary and intermediate features in the backbone network are fixed during the model training process. The meteorological processing module is used to input the meteorological element data into the meteorological feature extraction branch based on multilayer perceptron in the pre-trained visibility estimation model to obtain meteorological feature vectors. The feature fusion module is used to fuse the image feature vector and the meteorological feature vector to obtain fused features; The classification decision module is used to determine the current visibility level based on the fused features using a classifier.

9. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the visibility monitoring method based on image deep learning as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, can implement the visibility monitoring method based on image deep learning as described in any one of claims 1-7.

Images