Fog visibility detection method and device based on depth and brightness information, and medium

Abstract

A haze visibility detection method and device based on depth and brightness information and a medium are used for haze visibility detection, and a multi-modal visibility detection model utilizing image data and meteorological data is provided.The model simultaneously takes the image data and the meteorological data as inputs, and through a parallel network structure, the data of the two modalities are respectively sent into different channels to extract respective features, and then the features are fused to jointly invert the visibility value.The present application combines visual information and environmental parameters by constructing a multi-modal visibility detection model, introduces depth information and brightness information to optimize the model design, improves the accuracy and robustness of the model, and can more accurately and reliably detect the visibility.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross field of meteorology and computer vision, and relates to an image processing and deep learning technology, and is a fog and haze visibility detection method, device and medium based on depth and brightness information. BACKGROUND

[0002] In the field of meteorology, visibility is defined as the maximum distance that a person with normal vision can see a black object during the day or identify a light-emitting object at night, and fog and haze are the main factors affecting visibility. In the field of aviation, visibility is further defined as runway visual range, which is the maximum distance that a pilot can see runway markings or runway edge lights or centerline lights when the aircraft is on the runway centerline. Visibility is a key standard for determining whether an aircraft can take off and land safely, so accurate visibility detection is of great significance for flight planning and operation.

[0003] At present, most airports rely on professional visibility detection equipment such as scattering visibility meters, transmission visibility meters and laser radar visibility meters for visibility measurement. These devices are based on optical principles and calculate visibility by measuring optical parameters. However, these instruments are usually expensive, have high detection costs, and have strict requirements for the layout environment.

[0004] With the development of computer vision and deep learning technology, intelligent analysis methods based on video images have been increasingly applied in visibility detection. However, existing visibility detection models based on video images have many problems, such as: 1) only using general network models to extract image features, these network models have certain feature extraction capabilities and can be well applied to general target recognition or detection tasks, but visibility is different from target detection, and such general network models lack targeted design in capturing visibility-related feature information; 2) only performs well in daytime scenarios, but in nighttime scenarios, the detection accuracy of the model decreases significantly due to poor lighting conditions and insufficient image information. Therefore, there is still much room for improvement in existing visibility detection technology. SUMMARY

[0005] The problem to be solved by the present application is that existing visibility detection methods do not perform well in fog and haze conditions, and there is a need to improve the detection accuracy of visibility in fog and haze scenarios.

[0006] The technical solution of the present application is a fog and haze visibility detection method based on depth and brightness information for visibility detection in fog and haze conditions, comprising the following steps:

[0007] S1: Collect image data under different visibility conditions for the visibility scene to be measured, and simultaneously collect meteorological data and true values of visibility corresponding to the image time in parallel to construct a data set;

[0008] S2: Construct a visibility detection model, including an image data source channel, a meteorological data source channel, and a feature fusion module. The image data source channel and the meteorological data source channel are in parallel structure. Image data and meteorological data are taken as inputs at the same time. Through the parallel network structure, the data of the two modes are sent into different channels to extract their respective features. Then, the features of the two modes are fused through the feature fusion module to jointly invert and obtain the visibility value.

[0009] Specifically, when extracting features from image data source channels, the ViT model is used as the backbone network, and depth information is introduced into the self-attention mechanism of the ViT model to obtain a depth information-driven layer-by-layer self-attention mechanism, DDL Self-Attention. The specific feature extraction from image data source channels is as follows:

[0010] Obtain the depth map of the image. The depth map is divided into P image patches according to the ViT model. For each image patch, the depth value d... i The depth-weighted factor is obtained by performing an exponential transformation. i is the image patch index, and β is a hyperparameter that controls the weight amplification.

[0011] depth weighting factor w of all image patches depth ∈R P Constructed as a depth-weighted matrix W depth ∈R P×P Each row of the matrix is ​​w depth :

[0012]

[0013] w depth =[w depth,1 ,w dept h,2,……,w depth,P ]

[0014] In the stacking of layer-by-layer encoders in the ViT model, a dynamic adjustment factor α is introduced. l Simulating the change in attention during human visual perception of visibility, the image patches in the distance gradually gain more attention in deeper layers. The dynamic adjustment factor α of the l-th layer encoder... l for:

[0015]

[0016] Where L is the total number of encoder layers, l represents the current encoding layer index, and γ is a hyperparameter that controls the dynamic adjustment factor;

[0017] The depth weighting matrix of the l-th layer encoder is obtained as follows:

[0018] Wdepth,l = a l · W depth

[0019] The self-attention calculation result is adjusted based on the depth weighting matrix in each layer encoder, and the self-attention mechanism DDL Self-Attention of each layer encoder is as follows:

[0020]

[0021] Wherein, Q, K and V represent query, key and value vectors respectively, T represents transposition, d k is the embedding vector dimension, is the attention score matrix, W depth,l is the depth weighting matrix;

[0022] Based on DDL Self-Attention, the calculation process of extracting image features by the encoder of the ViT model is as follows:

[0023] Z′ l = DDL MSA (LN (Z l-1 ))+ Z l-1 l = 1, 2, …… L

[0024] Z l = MLP (LN (Z′ l ))+ Z′ l l = 1, 2, …… L

[0025] Wherein, LN represents layer normalization, Z l-1 represents the input of the lth layer encoder, DDL MSA represents the DDL Self-Attention mechanism, Z′ l represents the output of the intermediate layer of the encoder, MLP represents the feedforward layer, and Z l represents the output of the lth layer encoder.

[0026] After the stacking of the multi-layer ViT encoder, the output of the last layer ViT encoder is obtained through global average pooling to obtain the final global feature representation:

[0027] F image = GlobalAveragePooling (Z L )

[0028] Wherein, Z L represents the feature vector output by the last layer ViT encoder, and F image is the extracted image feature.

[0029] S3: training, verifying and testing the constructed visibility detection model based on the dataset;

[0030] S4: Model deployment, inputting real-time images for visibility detection.

[0031] Furthermore, in step S2, when fusing the features extracted from the image data and meteorological data through the feature fusion module, an adaptive weighted feature fusion strategy based on brightness information is adopted. This strategy dynamically weights the image and meteorological factor features by considering brightness information. Specifically:

[0032] First, the light intensity of the current image is determined by calculating the brightness value of the nighttime image. The image brightness B is obtained by calculating the average brightness after grayscale processing of the image.

[0033]

[0034] Where n represents the nth pixel in the image, Gray(n) represents the gray value of that pixel, N is the total number of pixels in the image, and the brightness value B reflects the overall illumination intensity of the image, with a value range of 0-255.

[0035] Based on the brightness information B, weighted coefficients for image features are generated using the Sigmoid mapping function:

[0036]

[0037] Among them, W image W is the image feature weighting coefficient, ranging from 0 to 1. U is a set brightness threshold hyperparameter. When the brightness value B is higher than the brightness threshold U, the weighting coefficient W... image When the brightness value B is close to 1, and the brightness threshold U is lower than the brightness value B, the weighting coefficient W is close to 1. image Approaching 0;

[0038] Based on the calculated weight coefficients, the image features and meteorological features are weighted and fused:

[0039] F fusion =W image ·F image +(1-W image )·F weather

[0040] Among them, F image For image features, F weather As a meteorological factor characteristic, F fusion For the weighted fusion features, F fusion Input the regression head to perform visibility prediction.

[0041] The application further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the visibility detection model of the fog and haze visibility detection method based on depth and brightness information.

[0042] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores at least one instruction or at least one program, and the at least one instruction or program, when executed, realizes the visibility detection model of the fog and haze visibility detection method based on depth and brightness information.

[0043] The fog and haze image and the visibility detection task are characterized by capturing the distribution of fog rather than common target recognition and detection, so that the general visual recognition model in the prior art cannot well adapt to this task, and the application proposes the design of DDL Attention after fully considering the characteristics of the fog and haze image.

[0044] The application has the following beneficial effects:

[0045] (1) An airport fog and haze visibility detection method based on depth and brightness information is proposed, the visual information is combined with the environmental parameters by constructing a multi-modal visibility detection model, especially the depth information and the brightness information are introduced to optimize the model design, the visibility can be more accurately and reliably detected, the existing detection method fails to fully utilize these information, the two kinds of information are comprehensively used in visual detection and the fusion of visual and meteorological detection, and the detection accuracy is effectively improved.

[0046] (2) When observing the visibility by the human eye, firstly, whether the fog and haze exists in the scene and the size of the fog and haze are roughly estimated, then the attention is gradually transferred from the close view in the scene to the distant view, and more attention is given to the distant view area, and the farthest area that can be recognized in the scene is tried to find, so that the size of the visibility is further judged, therefore, the distant view area in the fog and haze image can better reflect the actual visibility size. The application starts from the perspective of observing the fog and haze visibility by the human eye, and designs a depth information driven layer-by-layer self-attention mechanism DDL Self-Attention based on the benchmark model ViT, so as to better extract the visibility related features. The DDL Self-Attention effectively utilizes the depth information, and gradually increases the attention weight of the distant view area in the image, this way simulates the attention transfer process from near to far when the human perceives the fog and haze visibility, and gradually enhances the attention to the distant view area, so that the model can extract more effective feature information and better extract the visibility features, rather than paying too much attention to invalid information such as the foreground area.

[0047] (3) Considering that the night visibility detection is directly related to the light source, the worse the night illuminance is, the less effective feature information the image can provide, the present application proposes an adaptive weighted feature fusion strategy based on brightness information, which uses image brightness information as prior knowledge to dynamically weight the image and meteorological factor features, optimizes the image and meteorological factor feature fusion under different night light conditions, and improves the accuracy and robustness of the model in the night environment. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flowchart of the haze visibility detection method in the present application example.

[0049] Figure 2 The structural schematic diagram of the visibility detection model in the present application example.

[0050] Figure 3 The DDL Attention structural schematic diagram in the present application example.

[0051] Figure 4 The vision transformer encoder structural schematic diagram in the present application example.

[0052] Figure 5 The feature importance analysis schematic diagram in the present application example. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application more clear, specific embodiments will be further described in combination with the drawings and specific embodiments.

[0054] As shown in Figure 1 The present application provides an airport haze visibility detection method based on depth and brightness information, which comprises the following steps:

[0055] Step S1: Collect historical data and build a data set.

[0056] Image data under different visibility conditions of the airport is collected, and meteorological data and visibility true value corresponding to the image time are collected simultaneously. Among them, the image data is obtained by monitoring video shot by high-definition camera deployed in the airport, the meteorological data is obtained by automatic weather observation (AWOS) system deployed in the airport, which records a large number of meteorological elements including temperature, wind speed, precipitation, etc., and records once every minute, in order to keep the image data and meteorological data aligned and synchronized, one frame of image is extracted from the video every minute, and the visibility true value is obtained by the visibility detector deployed in the airport, finally a multi-modal data set is made, and the data set is divided into training set, validation set and test set according to the ratio of 7:2:1.

[0057] Step 2: Build a visibility detection model

[0058] As Figure 2 shown, the visibility detection model constructed by the present application is composed of four parts: an image data source channel, a meteorological data source channel, feature fusion, and a regression head. The image data source channel and the meteorological data source channel are parallel structures, respectively inputting data of two modalities of images and meteorological factors into a parallel backbone network to extract respective features, then fusing the features of the two modalities, and finally inputting the fused features into the regression head to jointly invert the visibility value.

[0059] In the image data source channel, first, the original input image is sharpened, and then the image features are extracted through the backbone network.

[0060] Among them, the image sharpening adopts a non-sharpening mask method, first smoothes the original image to obtain a blurred image, then calculates the difference between the original image and the blurred image to obtain a detail image, and finally adds the detail image back to the original image by a certain proportion, thereby enhancing the edges and details in the image. Through sharpening, the edge information in the image can be significantly enhanced, making the outlines and details of objects in the image clearer, highlighting the foreground and background in the image and the boundaries of foggy and non-foggy areas, and improving the overall visual quality of the image.

[0061] Then, the sharpened image is input into the backbone network to extract image features. The present application adopts a VisionTransformer model architecture as the backbone network of the image channel, and designs a depth information driven layerwise self-attention mechanism (DDL Self-Attention) to better capture the visibility features.

[0062] Specifically, when a human perceives the visibility of a scene, first, a rough estimate is made of whether there is fog and haze in the scene and the size of the fog and haze, then the attention gradually shifts from the close-up to the long shot in the scene, and more attention is given to the long shot area, trying to find the farthest area in the scene that can be recognized to further judge the size of the visibility. Therefore, the clarity of the long shot area in the image is a key factor in determining the size of the visibility, and the information of the long shot area can better reflect the actual farthest visible distance, while the clarity of the close-up area is less affected by the change of visibility. Therefore, more attention should be paid to the features of the long shot area in the image in the visibility detection task.

[0063] The ViT model divides the input image into n fixed-size image blocks and captures the dependency between the image blocks through a global self-attention mechanism in the encoder, and then generates the final image features through the stacking of multiple layers of encoders. However, the global self-attention mechanism may pay more attention to the foreground area with clearer and more obvious features in the calculation process, but in a certain scene with different haze densities, the changes in the foreground area are not obvious, so that the image features finally extracted by the network cannot fully represent different haze densities.

[0064] To this end, the application designs a depth information driven layer-by-layer self-attention mechanism (DDL Self-Attention) to simulate the attention shift process of human perception of haze visibility, as shown in Figure 3 By introducing depth information and dynamically adjusting the depth weighting factor layer by layer, the attention to the distant area is gradually enhanced, the attention shift conforming to human visual perception is realized, and more rich and effective visibility features are captured. The calculation steps of DDL Self-Attention are as follows:

[0065] For an original image I, first sharpen the image, and then obtain the corresponding depth map through the monocular depth estimation model Depth Anything V2:

[0066] D = Depth Anything V2(S)

[0067] Where S is the sharpened image, and D is the obtained depth map, the value range of the depth map is 0-1, representing the relative depth, close to 0 indicating the near view area, and close to 1 indicating the distant view area.

[0068] The depth map D is also divided into P fixed-size image blocks, and the depth value of each image block is represented by the average value of the depth values of all pixels in the image block:

[0069]

[0070] Where i is the image block index, M represents the total number of pixels in the image block, and D i,m represents the depth value of the mth pixel in the ith image block.

[0071] The depth value d i of each image block is exponentially transformed to obtain the depth weighting factor:

[0072]

[0073] Where d i is the depth value of the ith image block, and β is a hyperparameter that controls the weight amplification.

[0074] There are P image blocks in total, and the depth weighting factor w depth ∈R P is constructed as a depth weighting matrix W depth ∈R P×P Each row of which is w depth :

[0075]

[0076] w depth = [w depth,1 , w dept h,2, …, w depth,P ]

[0077] Here, the depth weighting factor is expanded into a matrix form to match the dimension of the original attention matrix, so that the depth weighting information can be applied to all image blocks in parallel in the process of calculating attention.

[0078] In the stack of layer-by-layer encoders, a dynamic adjustment factor a l Simulates the change process of attention when human visual perception visibility, so that the image blocks of the long view gradually get more attention in the deep layers of the stacked encoder. The dynamic adjustment factor of the l-th layer encoder is:

[0079]

[0080] Where L is the total number of encoders, l represents the current encoding layer index, and γ is a hyperparameter that controls the dynamic adjustment factor.

[0081] The depth weighting matrix of the l-th layer encoder can be obtained as:

[0082] W depth,l = a l · W depth

[0083] Adjust the original self-attention calculation result based on the depth weighting matrix in each layer encoder, and the calculation formula of DDL Self-Attention in each layer encoder is as follows:

[0084]

[0085] Where Q, K, and V represent query, key, and value vectors, T represents transposition, d k is the embedding vector dimension, is the attention score matrix, and W depth,l is the depth weighting matrix.

[0086] The corresponding ViT encoder structure is as follows: Figure 4As shown, the calculation process of extracting image features by the ViT encoder is as follows:

[0087] Z' l = DDL MSA(LN(Z l-1 ))+ Z l-1 l = 1, 2, …, L

[0088] Z l = MLP(LN(Z' l ))+ Z' l l = 1, 2, …, L

[0089] wherein LN represents layer normalization, Z l-1 represents the input of each layer of the encoder, DDL MSA represents the DDL multi-head self-attention mechanism of the application, Z' l represents the output of the intermediate layer of the encoder, MLP represents a feedforward layer, and Z l represents the output of the encoder.

[0090] After stacking the multi-layer ViT encoder, the output of the last layer of the ViT encoder is obtained through global average pooling to obtain the final global feature representation:

[0091] F image = GlobalAveragePooling(Z L )

[0092] wherein Z L represents the feature vector output by the last layer of the ViT encoder.

[0093] In this example, the input image size is fixed at 224x224x3, the image block size is 16x16, and the embedding dimension is 768. Therefore, the encoder input vector dimension is 196x768, and the vector dimension remains unchanged after processing by the multi-layer ViT encoder. The feature vector output by the last layer of the ViT encoder is obtained through the global average pooling layer, and the 768-dimensional feature is the feature representation of the image mapped to the visibility value.

[0094] In the meteorological data source channel, first, the key meteorological factors are extracted through feature importance analysis of diversified meteorological factors, and then the meteorological features are extracted through the backbone network.

[0095] The meteorological instrument of the airport records multiple meteorological factors, and the meteorological factor that can significantly affect the visibility is referred to as a key meteorological factor. The random forest is used to extract the key meteorological factor through feature importance analysis, and the calculation method is as follows:

[0096]

[0097] where node denotes a node in the decision tree, nodes denotes a set of all nodes in the decision tree, Δd(node) denotes the reduction of impurity brought by the feature f at the node node, S f,r denotes the importance score of the feature f on the rth decision tree, R is the number of decision trees, S f is the average score of the feature f on all trees, which is taken as the feature importance score. The feature importance scores of each meteorological factor are sorted, and several types of meteorological factors with high scores are selected as key meteorological factors.

[0098] The feature importance analysis result is shown in Table 1. Figure 5 The present application selects several meteorological factors that most obviously affect the visibility value, which are horizontal wind speed (WS2A), temperature (TEMP), wind direction (WD2A), humidity (RH), vertical wind speed (CW2A), and air pressure (PAINS).

[0099] The extraction of meteorological features uses a fully connected neural network. Before inputting the key meteorological factors into the backbone network, the data is standardized by using the z-score standardization method to eliminate the dimensional differences between different meteorological factors:

[0100]

[0101] where x j is the jth meteorological factor feature, μ j is the mean of the sample data of the meteorological factor, σ j is the standard deviation of the sample data of the meteorological factor, and x j,norm is the standardized value of the meteorological factor.

[0102] The input data contains J meteorological factors, which is represented as a vector x ∈ R J . The neurons of each layer first perform linear transformation on the input. For the hth layer, the linear transformation formula is:

[0103] z (h) = W (h) a (h-1) +b (h)

[0104] where W (h) is the weight matrix of the hth layer, a (h-1) is the input of the h-1th layer, b (h) is the bias of the hth layer, and z (h) is the linear transformation result of the hth layer. Then the linear transformation result is converted into a nonlinear output through an activation function, where act() is the activation function:

[0105] a(h) = act(z (h) )

[0106] Assuming the last hidden layer is the H layer, the output a (H) is the feature representation of the weather data mapped to the visibility value:

[0107] F weather = a (H)

[0108] In the examples of the present application, the input is 6 weather factors, so the number of neurons in the input layer of the fully connected neural network is 6, the hidden layer is set to two layers, and in order to balance the effects of the two modal data, the feature dimensions of the two modalities are kept consistent, so the number of neurons in the first layer is set to 512, and the number of neurons in the second layer is set to 768. The 768-dimensional features output by the second hidden layer are used as the feature representation of the weather factors mapped to the visibility value.

[0109] Further, the image features and weather features are fused to jointly invert the visibility value based on the two modalities.

[0110] Considering that the visibility detection at night is greatly affected by the environmental light, when the light intensity at night is high, the image can provide clear feature information for effective visibility detection. However, when the light is weak or there is no light source, the night image often lacks sufficient contrast and details, and cannot provide effective visibility information, especially in the long-range area of the image, so that the model is difficult to extract key features related to visibility from the image. At this time, the visibility detection through the weather factor is more accurate and effective. Therefore, the present application adopts an adaptive weighted feature fusion strategy based on brightness information, which dynamically weights the image and weather factor features by considering the brightness information, thereby optimizing feature fusion under different environmental light and improving the accuracy of visibility detection. The steps are as follows:

[0111] First, the brightness value of the night image is calculated to determine the light intensity of the current image. The brightness B of the image can be obtained by calculating the average brightness after grayscale processing of the image:

[0112]

[0113] Where n represents the nth pixel in the image, Gray(n) represents the gray value of the pixel, N is the total number of image pixels, and the brightness value B reflects the overall light intensity of the image, with a value range of 0-255.

[0114] Based on the brightness information B, the weighting coefficient of the image feature is generated by the Sigmoid mapping function:

[0115]

[0116] wherein W image is an image feature weight coefficient, the value range is 0-1, U is a set brightness threshold hyperparameter, when the brightness value B is higher than the brightness threshold U, the weight coefficient W image is close to 1, when the brightness value B is lower than the brightness threshold U, the weight coefficient W image is close to 0.

[0117] According to the calculated weight coefficient, the image feature and the weather feature are weighted and fused:

[0118] F fusion = W image · F image + (1-W image )· F weather

[0119] wherein F image is an image feature, F weather is a weather factor feature, and F fusion is a weighted fused feature.

[0120] Through the feature fusion strategy based on brightness information, the brightness information is used as prior knowledge to guide the feature fusion, helping the model to learn. This strategy can better cope with the changes of environmental light conditions in night visibility detection, and improve the accuracy of night scene visibility detection.

[0121] Finally, the attention weighted feature representation is input into the regression head for the final visibility prediction. In this example, the regression head consists of two fully connected layers and a final output layer, with neuron numbers of 1024 and 512 respectively. Since the final visibility value is to be predicted, the output layer has one neuron.

[0122] In step S3, the dataset constructed in step S1 is input into the visibility detection model constructed in step S2 for model training, verification and testing. The loss function used in training and verification is the mean square error function, which is calculated as follows:

[0123]

[0124] wherein y t is the true value of the tth sample, y is the predicted value of the tth sample, and Y is the sample number.

[0125] The optimizer uses the AdamW optimizer, the learning rate is set to 0.0001, the loss is calculated with respect to the entire model parameters by back propagation, and then the model parameters are updated to minimize the loss until the model converges. After the end of each training cycle, the model is applied to the validation set, and the model hyperparameters are adjusted to optimize the training effect. Finally, the best-performing model on the validation set is saved, and the model is finally evaluated using the test set data. By continuously optimizing the model parameters, the accuracy and robustness of the model are improved, ensuring that the model can accurately detect visibility under various conditions.

[0126] In step S4, the visibility detection model trained and tested in step S3 is deployed to the scene to be detected, such as an airport monitoring system, to obtain real-time picture data of the monitoring camera and input it into the model for visibility detection. The system can feed back the current visibility situation in real time according to the output of the model, assisting the operation decision of the airport.

[0127] The present application can be implemented based on a computer program, and therefore the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor, and the visibility detection model of the above-mentioned haze visibility detection method based on depth and brightness information is used for real-time detection of haze visibility. A computer readable storage medium is also provided, wherein the computer readable storage medium stores at least one instruction or at least one program, and the at least one instruction or program is executed to implement the visibility detection model of the above-mentioned haze visibility detection method based on depth and brightness information. The instructions or program codes for implementing the method of the present application can be written in any combination of one or more programming languages. The instructions or program codes can be executed completely on a data processor, partially on a processor, partially on a processor as a separate software package, and partially on a remote device or a server.

Claims

1. A method for detecting visibility in haze based on depth and brightness information, used for visibility detection in haze conditions, characterized by: Includes the following steps: S1: Collect image data of the visibility scene to be tested under different visibility conditions, and simultaneously collect meteorological data corresponding to the image time and the true visibility value to construct a dataset. S2: Construct a visibility detection model, including an image data source channel, a meteorological data source channel, and a feature fusion module. The image data source channel and the meteorological data source channel are in parallel structure. Image data and meteorological data are taken as inputs at the same time. Through the parallel network structure, the data of the two modes are sent into different channels to extract their respective features. Then, the features of the two modes are fused through the feature fusion module to jointly invert and obtain the visibility value. Specifically, when extracting features from image data source channels, the ViT model is used as the backbone network, and depth information is introduced into the self-attention mechanism of the ViT model to obtain a depth information-driven layer-by-layer self-attention mechanism, DDL Self-Attention. The specific feature extraction from image data source channels is as follows: Obtain the depth map of the image. The depth map is divided according to the processing of the ViT model. For each image patch, the depth value is... Perform an exponential transformation to obtain the depth-weighted factor. , For image patch indexing, To control the hyperparameters of weight amplification; Depth weighting factor for all image patches Constructed as a depth-weighted matrix Each row of the matrix is : In the stacking of layer-by-layer encoders in the ViT model, a dynamic adjustment factor is introduced. Simulating the changes in attention during human visual perception of visibility, this method allows distant image patches to gradually gain more attention in deeper layers. Dynamic adjustment factor of layer encoder for: in, This represents the total number of layers in the encoder. Indicates the index of the current coding layer. It is a hyperparameter that controls the dynamic adjustment factor; Get the first The depth weighting matrix of the layer encoder is: In each encoder layer, the self-attention calculation results are adjusted based on the depth-weighted matrix. The self-attention mechanism (DDL) for each encoder layer is as follows: in, These represent the query, key, and value vectors, respectively. Indicates transpose. For the embedded vector dimension, The attention score matrix, It is a depth-weighted matrix; The computational process for extracting image features using the encoder of the ViT model based on DDL Self-Attention is as follows: in, Representation layer normalization, Indicates the first The input of the layer encoder, This indicates the DDL Self-Attention mechanism. This represents the output of the encoder's intermediate layer. Indicates feedforward layer, Indicates the first The output of the layer encoder; After stacking multiple layers of ViT encoders, the output of the last ViT encoder is obtained by global average pooling to obtain the final global feature representation: in This represents the feature vector output by the last ViT encoder layer. That is, the extracted image features; S3: Train, validate, and test the constructed visibility detection model based on the dataset; S4: Model deployment, inputting real-time images for visibility detection.

2. The haze visibility detection method based on depth and brightness information according to claim 1 is characterized in that, in step S1, image data is collected by monitoring video of the visibility scene to be tested, and one frame is extracted from the video every minute to keep the image data and meteorological data aligned and synchronized. The true visibility value is obtained by deploying a visibility detector, and the constructed dataset is divided into training set, validation set and test set in a ratio of 7:2:

1.

3. The haze visibility detection method based on depth and brightness information according to claim 1, characterized in that: Before feature extraction from the image data source channel, image sharpening is performed using an unsharpened masking method.

4. The haze visibility detection method based on depth and brightness information according to claim 1, characterized in that: The meteorological data source channel first performs feature importance analysis on diverse meteorological factors to extract key meteorological factors, and then extracts meteorological features through the backbone network, specifically: For meteorological factors recorded by weather stations, those that significantly affect visibility are called key meteorological factors. Random forest is used for feature importance analysis, and the calculation method is as follows: in, Represents the nodes in the decision tree. This represents the set of all nodes in the decision tree. Representation of features At the node The reduction in impurities brought about by the above. Representation of features In the The importance score on each tree It is the number of decision trees. It is a feature The average score across all trees is used as the feature importance score. The feature importance scores of each meteorological factor are ranked, and the meteorological factors with the highest scores are selected as key meteorological factors. Meteorological features were extracted using a fully connected neural network. Before inputting key meteorological factors into the backbone network, the data was standardized using the z-score normalization method to eliminate differences in the units of measurement between different meteorological factors. in, For the j-th meteorological factor characteristic, This represents the mean of the sample data for this meteorological factor. The standard deviation of the sample data for this meteorological factor is... This is the standardized value of the meteorological factor. The standardized key meteorological factors are then input into a fully connected neural network to extract features. Assuming the input contains J meteorological factors, represented as a vector... Each layer of neurons first performs a linear transformation on the input. For the first layer... The linear transformation formula for the layer is: in, It is the first The weight matrix of the layer, It is the first Layer input, It is the first Layer bias, It is the first The linear transformation result of the layer is then converted into a nonlinear output through an activation function, where... For activation function: Assuming the last hidden layer is the Hth layer, its output... Feature representation of meteorological data mapped to visibility values: 。 5. The haze visibility detection method based on depth and brightness information according to claim 1, characterized in that, in step S2, when fusing the features extracted from image data and meteorological data through the feature fusion module, an adaptive weighted feature fusion strategy based on brightness information is adopted, which dynamically weights image and meteorological factor features by considering brightness information, specifically: First, the light intensity of the current image is determined by calculating the brightness value of the nighttime image. The average brightness is obtained by calculating the image after grayscale conversion. in, Indicates the first in the image 1 pixel, This represents the grayscale value of the pixel. It is the total number of pixels in the image, and the brightness value. It reflects the overall illumination intensity of the image, with a value range of 0-255; Based on brightness information The weighted coefficients of image features are generated using the Sigmoid mapping function: in, These are image feature weighting coefficients, with values ​​ranging from 0 to 1. It is the set brightness threshold hyperparameter; when the brightness value... Above the brightness threshold At that time, weighting coefficient When the brightness value is close to 1 Below the brightness threshold At that time, weighting coefficient Approaching 0; Based on the calculated weight coefficients, the image features and meteorological features are weighted and fused: in, For image features, Characteristics of meteorological factors, For the weighted fusion features, Input the regression head to perform visibility prediction.

6. An electronic device, characterized in that: The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the visibility detection model of the haze visibility detection method based on depth and brightness information as described in any one of claims 1-5, and is used to detect haze visibility in real time.

7. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the visibility detection model of the haze visibility detection method based on depth and brightness information as described in any one of claims 1-5.

Images