Deep learning-based regale visibility estimation method and system, and medium

Abstract

The invention discloses a regale visibility estimation method and system based on deep learning, and a medium, and the method comprises the steps: inputting a picture into a residual network module and a joint attention module, dividing the picture into a plurality of blocks, inputting the blocks into the joint attention module, and enabling each block to be a local image sequence; feature fusion is carried out, and feature mapping from the convolutional layer is integrated; finally, processing is carried out through a normalization layer and an average pooling layer in sequence, normalization is carried out on each feature channel of each sample through the normalization layer, and the spatial dimension of the features is reduced through the average pooling layer; and finally, sending to a Head layer, and estimating the visibility grade. Under the influence of different illuminance, temperature, rain and fog and other factors, the visibility level can be accurately estimated for river surface images with different degrees of water vapor, the influence and misjudgment caused by existence of river bank vegetation, staying ships and non-uniform water vapor on the river surface can be avoided to the maximum extent, and the method has good practicability.

Description

Technical Field

[0001] The present invention belongs to the technical field of visibility estimation based on deep learning, and specifically relates to a Minjiang River visibility estimation method, system and medium based on deep learning. Background Art

[0002] Visibility is a descriptive indicator of visual obstacles caused by weather in the field of roads and navigation. River surface visibility has a significant impact on navigation safety, environmental quality assessment, climate research, etc., so it is particularly necessary to carry out river surface visibility detection. For visibility detection in various scenes, the current traditional visibility detection methods include visual inspection, instrument method and visibility estimation method based on physical model.

[0003] However, the visual method cannot carry too much data when facing scenes that require real-time detection, and is greatly affected by the subjective influence of the observer. The instrument method uses optical and electromagnetic instruments to detect relevant parameters in the air, thereby completing the calculation of visibility. The existing instruments mainly include atmospheric transmission instruments and forward scattering instruments. The principle is to measure the atmospheric transmittance or extinction coefficient through sensors to calculate the current environmental visibility. For the time being, it can only achieve visibility detection at short distances. It is mainly used for local detection in the meteorological field. Moreover, such equipment is very expensive and not suitable for daily large-scale installation. With the development of computer vision, many scholars have started to work on visibility detection based on physical models. For example, Bronte et al. used the Sobel operator for edge detection, and Mao et al. combined the atmospheric scattering model to evaluate dense fog. However, these methods have high requirements for scenes. For example, obstacles caused by the presence of cluster fog, riverbank vegetation, and uneven water vapor on the river surface will affect the extraction of physical models.

[0004] In order to solve this series of problems, the deep learning-based method has shown great advantages. It can input large-scale data and let the computer network learn the complex relationship between images. However, in different scenarios, river surface images lack "specific features", and the visibility estimation method based on deep learning also has the problem of incomplete feature extraction leading to low accuracy.

[0005] The fog on the river surface is usually accompanied by special weather factors. Due to the influence of multiple factors such as illumination, temperature, rain and fog, different degrees of water vapor are often produced. In the river surface observation image, the river surface, the fog junction area and the sky area are difficult to distinguish, and there are interference factors such as riverbank vegetation and parked ships at the edge of the image. In the Minjiang River image, the presence of riverbank vegetation, water vapor on the river surface and weather influences have a significant impact on the visual observation of the water surface. The presence of trees may cause visual occlusion and reflection interference, while the water vapor on the river surface increases the change in atmospheric transmittance and reduces visibility. If extreme weather such as heavy fog and hail is encountered, it will more seriously affect the accurate assessment of the river surface conditions.

[0006] If we want to better observe the visibility of the river surface based on deep learning, we need to learn more detailed spatial information. Most general deep learning classification or linear regression models rely on CNN, but the learning ability of CNN has a certain upper limit. The robustness and generalization of visibility detection are poor. When the accuracy of data fitting reaches a certain stage, more training data is needed to improve the corresponding estimation accuracy. Therefore, it is still challenging to accurately detect the precise visibility value or visibility level. Summary of the invention

[0007] The purpose of the present invention is to provide a method, system and medium for estimating the visibility of the Minjiang River based on deep learning, aiming to solve the above problems. The present invention solves the problem that the current visibility estimation method based on deep learning has low accuracy and the estimation effect of the same model on different data sets has large differences.

[0008] The present invention is mainly achieved through the following technical solutions:

[0009] A method for estimating visibility of the Minjiang River based on deep learning comprises the following steps:

[0010] Step S1: Input the pictures into the residual network module in sequence, use the residual network to extract the feature information of the pictures, and then dynamically adjust the response strength of the feature map based on the spatial information of the attention module;

[0011] Step S2: Divide the input image into several blocks, and each block is a local image sequence; input the divided image into a joint attention module, so as to extract features in each local image sequence through the self-attention mechanism, and then capture the dependency between images through cross-grouping;

[0012] Step S3: The output information of step S1 and step S2 is subjected to feature fusion, and the feature map from the convolution layer is integrated; finally, it is processed by the normalization layer and the average pooling layer in sequence, each feature channel of each sample is normalized by the normalization layer, and the spatial dimension of the feature is reduced by the average pooling layer; finally, it is sent to the Head layer to estimate the visibility level.

[0013] In order to better implement the present invention, further, in the step S2, the divided image is processed from the first stage to the third stage in sequence. In the first stage and the second stage, the local information, neighbor information and global information of the feature map are interacted through the LNG Self-Attention module, and the height and width are downsampled once, and the feature channel dimension is doubled; in the third stage, the local information of the feature map is learned through the LL Self-Attention module.

[0014] In order to better implement the present invention, further, in step S1, the residual network is a ResNet18 network.

[0015] In order to better implement the present invention, further, in the step S3, at the Head layer, a flattening operation is performed and the features are sent to the fully connected layer to learn the mapping relationship between the features and the category labels to provide a decision basis for the classification task; finally, the learned features are mapped to the final output.

[0016] The present invention is mainly achieved through the following technical solutions:

[0017] A Minjiang River visibility estimation system based on deep learning is based on the above-mentioned Minjiang River visibility estimation method, and includes a picture acquisition module and an estimation module; the picture acquisition module is used to collect pictures of the Minjiang River surface, and the estimation module is used to input the collected pictures of the Minjiang River surface into the estimation network model to obtain the visibility level;

[0018] The estimation network model includes a residual network module, a joint attention module, a feature fusion layer and a level estimation module arranged in parallel; the residual network module and the joint attention module are respectively connected to the level estimation module through the feature fusion layer; the level estimation module is used to output the visibility level.

[0019] In order to better realize the present invention, further, the joint attention module includes a LNGSelf-Attention module, a LNG Self-Attention module and a LL Self-Attention module connected in sequence; the LNGSelf-Attention module includes a Local Self-Attention module, a NeighborSelf-Attention module and a Global Self-Attention module arranged in sequence from front to back, which are respectively used for interacting local information, neighbor information and global information; the LL Self-Attention module is composed of two Local Self-Attention modules connected in series, and only learns local information.

[0020] In order to better implement the present invention, further, the residual network module includes a CNN module and a CA module connected sequentially from front to back, and the CNN module is a ResNet18 network.

[0021] In order to better implement the present invention, further, the level estimation module includes a normalization layer, an average pooling layer and a Head layer which are arranged in sequence from front to back.

[0022] A computer-readable storage medium stores a computer program, which implements the above-mentioned Minjiang River visibility estimation method when executed by a processor.

[0023] The beneficial effects of the present invention are as follows:

[0024] The present invention achieves more accurate visibility estimation by combining the "engineered features" and "learned features" of the input image. Specifically, the present invention introduces an improved multi-level Transformer encoder, namely a joint attention module, to extract more image features. While interacting with local information, neighbor information and global information, the height and width are downsampled once, and the feature channel dimension is doubled, which is conducive to the network extracting features of different scales. The residual connection introduced at a shallower layer can better solve the gradient vanishing and gradient exploding problems in deep neural networks. A CA module is added after the residual to capture the relationship between channels, thereby improving the ability of feature representation. Therefore, under the influence of different illumination, temperature, rain and fog and other factors, the present invention can more accurately estimate the visibility level of river surface images with different degrees of water vapor, and can avoid the influence and misjudgment caused by riverbank vegetation, stationary ships and uneven water vapor on the river surface to the greatest extent.

[0025] The advantages of the estimation network model designed in the present invention are mainly: (1) The added CA module can capture long-range dependencies and save precise location information, while effectively using this information to accurately locate the area of ​​interest and capture the relationship between channels, thereby improving the ability of feature representation. (2) The LNG Self-Attention module of the improved Transformer hierarchy extracts information including local information, neighbor information and global information. This method can learn the image in more detail and extract more comprehensive information, and obtain higher accuracy with higher efficiency. (3) After the fusion of the Transformer-based self-attention mechanism and the convolutional neural network hierarchical structure, the network's focus area on the image pays more attention to the details. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of the structure of the ResNet18 network;

[0027] Figure 2 It is a schematic diagram of the structure of the LNG-Self-Attention module;

[0028] Figure 3 Schematic diagram of the structure of the estimation network model. DETAILED DESCRIPTION

[0029] Embodiment 1:

[0030] A method for estimating the visibility of the Minjiang River based on deep learning is used to predict visibility based on a constructed prediction network model. The prediction network model includes a joint attention module L-Attention and a residual network module based on the improved self-attention mechanism of Transformer. First, a convolutional neural network (CNN) is used to introduce residual connections to solve the gradient vanishing and gradient exploding problems in deep neural networks, and then an attention mechanism (Coordinate Attention, CA) module is added to achieve a deeper network structure and higher classification accuracy. Secondly, in the joint attention module L-Attention, the input image is divided into multiple small blocks by using its block strategy, each small block is a local image sequence, so that features are extracted from each local image sequence by the self-attention mechanism, and then the dependency between images is captured by cross grouping. Finally, after the Layer normal and AveragePooling operations, the image is sent to the Head layer for visibility estimation. The present invention can avoid the influence of river bank interference and pay more attention to the area of ​​interest when observing the river surface, so as to more accurately detect the visibility level of the river surface.

[0031] Preferably, if Figure 3 As shown, the estimation network model includes a residual network module and a joint attention module L-Attention, a feature fusion layer and a level estimation module arranged in parallel, wherein the residual network module and the joint attention module L-Attention are respectively connected to the level estimation module through the feature fusion layer; the level estimation module is used to output the visibility level. Specifically, the level estimation module includes a normalization layer, an average pooling layer and a Head layer arranged in sequence from front to back.

[0032] The joint attention module L-Attention includes an LNG Self-Attention module, an LNGSelf-Attention module and an LL Self-Attention module connected in sequence; the LNG Self-Attention module includes a Local Self-Attention module, a Neighbor Self-Attention module and a GlobalSelf-Attention module arranged in sequence from front to back, which are respectively used for interacting local information, neighbor information and global information; the LL Self-Attention module is composed of two Local Self-Attention modules connected in series and only learns local information.

[0033] The residual network module includes a CNN module and a CA module connected sequentially from front to back, and the CNN module is a residual network structure. The present invention enhances the model's ability to capture image detail information by fusing a convolutional neural residual network with an attention mechanism. The residual network is known for its residual connections in deep networks, which can effectively alleviate the gradient vanishing problem and improve the efficiency of model training. However, as the depth of the network increases, the model may ignore some important local detail information. To solve this problem, we introduced an attention mechanism. This mechanism emphasizes those areas that are more important for specific tasks by assigning different weights to each position in the feature map. Specifically, the attention module uses spatial information to dynamically adjust the response intensity of the feature map, so that the model can pay more attention to those areas containing rich information. In this way, the model can not only learn global context information, but also retain and strengthen those detail features that are critical for accurate classification or detection. Combining the residual network with CA, our model has shown excellent performance in multiple visual recognition tasks. Experimental results show that this innovative fusion strategy not only improves the model's sensitivity to details, but also enhances its generalization ability, thereby achieving higher accuracy and robustness in complex image recognition scenarios.

[0034] Specifically, the structure of the residual network module is as follows:

[0035] The CNN module of the present invention uses the classic residual network Resnet. Commonly used ResNet models include ResNet18, ResNet34, and ResNet50. The residual connection introduced by the Resnet series can better solve the gradient vanishing and gradient exploding problems in deep neural networks, thereby realizing a deeper network structure. Therefore, it also shows good capabilities in classification activities and can achieve a higher accuracy rate. In order to avoid unnecessary structuring to increase the complexity of the overall model, a Resnet18 network with fewer layers is selected.

[0036] like Figure 1As shown in the figure, the network structure of ResNet18 is mainly composed of input layer, Convolutional Layers, and Residual Blocks. The input layer accepts three-channel RGB images. Convolutional Layers contain 3*3 convolution kernels and 2*2 max-pooling layers. Residual Blocks is the core of Resnet18. The solution to the gradient disappearance and gradient explosion problems depends on this structure. It contains two convolutional layers and a jump connection. The jump connection can directly connect the input to the output. ResNet18 contains multiple Residual Blocks, including four shallower blocks and four deeper blocks. The superposition of these blocks can gradually increase the depth of the network.

[0037] Specifically, the structure of the improved joint self-attention module based on Transformer is as follows:

[0038] The joint self-attention module L-Attention is an improvement on Transformer. It contains 3 stages, such as Figure 2 As shown in the figure, in the first two stages, the input feature map will interact with local information, neighbor information and global information, namely the LNG Self-Attention module. At the same time, the height and width are downsampled once, and the feature channel dimension is doubled, which is conducive to the network to extract features of different scales; in the last stage, only local information learning is performed, namely the LL Self-Attention module, which is conducive to reducing network redundancy and avoiding over-extraction and learning. LNG

[0039] The combination of the Self-Attention module and the LL Self-Attention module is a joint self-attention module, which can effectively reduce the complexity of linear calculations and is lighter and faster than traditional Vsion Transformer and Swin Transformer. The structure of LL Self-Attention is composed of two Local Self-Attention modules in series, which means that only local information is learned.

[0040] The key components of the Transformer architecture are introduced to significantly improve the comprehensive perception of local features, neighboring features, and global context information. This improved network structure adopts the self-attention mechanism, which allows the model to not only focus on the details of the local area when processing the image, but also capture the relationship between pixels, thereby realizing the sensitivity analysis of the correlation features of the spatially neighboring areas. In addition, through the multi-head attention technology in the Transformer, the model can learn features from multiple angles and scales in parallel, which further enhances its ability to capture information at different levels. At the same time, the inter-layer connection (such as residual connection) and layer normalization strategy in the model effectively alleviate the gradient vanishing and gradient exploding problems in deep network training, ensuring the stability and performance of the deep structure. More importantly, the encoder-decoder structure in the Transformer architecture provides our model with a powerful global information modeling capability. The encoder part is responsible for extracting and encoding global context information, while the decoder part is responsible for combining this global information with local features for accurate image analysis and interpretation.

[0041] Specifically, the structure of the feature fusion and level estimation module is as follows:

[0042] The feature fusion layer is responsible for integrating the feature maps from the previous convolutional layer to enhance the expressiveness of the features. Subsequently, it is sent to the normalization layer (Layer normal) to normalize each feature channel of each sample, which helps to stabilize and accelerate the training process of the deep network, reduce internal covariate shift, and allow a higher learning rate, while improving the generalization ability of the model; after that, the pooling operation is performed to effectively reduce the number of parameters and computational complexity by reducing the spatial dimension of the features, while enhancing the robustness of the features. Finally, it is sent to the Head layer, and after a flattening operation, these features are sent to the fully connected layer. This layer provides a decision basis for the classification task by learning the mapping relationship between features and category labels, and maps the learned features to the final output. The classifier module outputs the probability prediction of each category. By defining a suitable loss function and applying the back-propagation algorithm, the network can continuously optimize its parameters during the training process to improve the accuracy of classification.

[0043] For the extracted deep features and shallow features, feature fusion aims to improve classification accuracy by integrating features at different levels. In deep convolutional neural networks (CNNs), low-level features usually contain more spatial detail information, while high-level features contain richer semantic information. By designing a feature fusion framework, the effective integration of these complementary information is achieved, thereby effectively improving classification accuracy.

[0044] In scenarios affected by complex factors such as different illumination, temperature, rain and fog, the present invention uses a convolutional neural residual network combined with an attention mechanism to obtain detailed information, so that the model focuses on shallow detail extraction. Secondly, the present invention uses a deep network structure to focus on local information, neighbor information, and global information features, and then performs feature fusion at different levels, extracting more and more detailed information and fusing it with shallow information, which helps the model to better estimate visibility and improve accuracy overall. As shown in Tables 1 and 2, compared with existing models, the present invention has the best top1-accuracy in data sets Data1 and Data2, and the corresponding accuracy and recall rates have achieved significant progress.

[0045] Table 1

[0046]

[0047] Table 2

[0048]

[0049] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for estimating visibility of the Minjiang River based on deep learning, characterized in that: The following steps are involved: Step S1: Input the pictures into the residual network module in sequence, use the residual network to extract the feature information of the pictures, and then dynamically adjust the response strength of the feature map based on the spatial information of the attention module; Step S2: Divide the input image into several blocks, and each block is a local image sequence; input the divided image into a joint attention module, so as to extract features in each local image sequence through the self-attention mechanism, and then capture the dependency between images through cross-grouping; Step S3: The output information of step S1 and step S2 is subjected to feature fusion, and the feature map from the convolution layer is integrated; finally, it is processed by the normalization layer and the average pooling layer in sequence, each feature channel of each sample is normalized by the normalization layer, and the spatial dimension of the feature is reduced by the average pooling layer; finally, it is sent to the Head layer to estimate the visibility level.

2. The method for estimating visibility of Minjiang River based on deep learning according to claim 1, characterized in that: In step S2, the divided images are processed from the first stage to the third stage in sequence. In the first stage and the second stage, the local information, neighbor information and global information of the feature map are interacted through the LNG Self-Attention module, and the height and width are downsampled once, and the feature channel dimension is doubled; in the third stage, the local information of the feature map is learned through the LL Self-Attention module.

3. The method for estimating visibility of Minjiang River based on deep learning according to claim 1, characterized in that: In step S1, the residual network is a ResNet18 network.

4. The method for estimating visibility of Minjiang River based on deep learning according to claim 1, characterized in that: In step S3, a flattening operation is performed at the Head layer and the features are sent to the fully connected layer to learn the mapping relationship between the features and the category labels to provide a decision basis for the classification task; finally, the learned features are mapped to the final output.

5. A Minjiang River visibility estimation system based on deep learning, based on the Minjiang River visibility estimation method according to any one of claims 1 to 4, characterized in that: It includes a picture acquisition module and an estimation module; the picture acquisition module is used to collect pictures of the surface of the Minjiang River, and the estimation module is used to input the collected pictures of the surface of the Minjiang River into the estimation network model to obtain the visibility level; The estimation network model includes a residual network module, a joint attention module, a feature fusion layer and a level estimation module arranged in parallel; the residual network module and the joint attention module are respectively connected to the level estimation module through the feature fusion layer; the level estimation module is used to output the visibility level.

6. The Minjiang River visibility estimation system based on deep learning according to claim 5, characterized in that: The joint attention module includes an LNG Self-Attention module, an LNG Self-Attention module and an LLSelf-Attention module connected in sequence; The LNG Self-Attention module includes a Local Self-Attention module, a Neighbor Self-Attention module and a Global Self-Attention module, which are arranged in sequence from front to back, and are used to interact with local information, neighbor information and global information respectively; the LL Self-Attention module is composed of two Local Self-Attention modules connected in series, and only learns local information.

7. The Minjiang River visibility estimation system based on deep learning according to claim 5, characterized in that: The residual network module includes a CNN module and a CA module connected sequentially from front to back, and the CNN module is a ResNet18 network.

8. The Minjiang River visibility estimation system based on deep learning according to claim 5, characterized in that: The level estimation module includes a normalization layer, an average pooling layer and a Head layer which are arranged in sequence from front to back.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the Minjiang River visibility estimation method described in any one of claims 1-4 is implemented.

Images