A multi-meteorological sea ice detection algorithm based on SAF-FCOS heterogeneous data fusion

Abstract

The present application relates to a kind of multi-weather sea surface floating ice detection algorithm based on SAF-FCOS heterogenous data fusion, it is related to computer vision artificial intelligence target detection technical field.The multi-weather sea surface floating ice detection algorithm includes the following steps: using heterogenous data, simulation generation dataset, data set is divided according to task demand;Add attention mechanism module;Feature fusion;Model training;And the sea ice of test set is carried out target detection using the trained network model.The multi-weather sea surface floating ice detection algorithm based on SAF-FCOS heterogenous data fusion of the present application, anchor-free based on SAF-FCOS, can use the data from different sensors under different weather conditions after registration for the detection of sea surface floating ice;According to actual situation, the model needed is selected, and the robustness of the whole project is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision artificial intelligence target detection, and particularly relates to a multi-weather sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion. BACKGROUND

[0002] Sea surface floating ice detection plays an important role in climate research, navigation safety, and disaster monitoring and early warning. Sea ice state is a basic parameter for describing the polar environment and is of great significance to climate change research. By monitoring sea ice changes, the understanding of the air-ice-ocean interaction mechanism can be improved, supporting climate research and forecasting work. Sea ice detection also provides important information support for governments in coastal areas, marine operating units, and others, helping to ensure navigation safety and respond to potential threats from sea ice in a timely manner. Through remote sensing monitoring and other technical means, the type and distribution of sea ice can be monitored in real time, improving the monitoring and early warning capabilities of marine disasters and reducing disaster losses.

[0003] Deep learning has been widely applied in the field of image target detection, especially the SAF-FCOS network, which has excellent performance in anchor-free single-stage networks. With the powerful self-learning ability of deep learning, the low efficiency of traditional object detection can be solved. However, the original SAF-FCOS network is designed for multi-class target detection, so it is not completely suitable for sea ice detection under multi-weather conditions. When the weather conditions are slightly complex, the recall rate and precision of the detection effect under a single channel will decrease significantly. The feature fusion module of the original network is only a simple multiplication of the corresponding pixel values. When detecting sea ice targets, due to the small scale, it cannot utilize multi-layer feature information, and the representation ability is low, resulting in partial false detection and missed detection. SUMMARY

[0004] The present application solves the technical problems in the prior art and provides a multi-weather sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion.

[0005] To solve the above technical problems, the technical solution of the present application is as follows:

[0006] A multi-weather sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion includes the following steps:

[0007] Step 1: Use heterogeneous data to simulate a data set, and divide the data set according to task requirements;

[0008] Step 2: Add an attention mechanism module;

[0009] Step 3: Feature fusion;

[0010] The feature vectors of different sources are stacked by using a fusion layer in the feature fusion module, and a convolution module in the feature fusion module is used to generate a feature vector of a specific channel number, so as to realize subsequent pixel-by-pixel multiplication fusion.

[0011] Step 4: model training;

[0012] Different data sources and weather conditions are trained to obtain optimal models suitable for various conditions.

[0013] Step 5: using the trained network model to detect the sea ice in the test set, obtaining the attribute, confidence and position information of the target and marking.

[0014] In the above technical solution, step 1 is specifically:

[0015] The image of the floating ice is simulated to generate images or radar message information of various sensors under different weather conditions.

[0016] The distance, echo intensity and weather condition in the radar message information are mapped to the image coordinate system according to certain rules, and input into the network for training.

[0017] In the above technical solution, the radar message data in step 1 is mapped to a series of small circles, and the calculation formula is:

[0018]

[0019]

[0020] dis=F(d rgb )echo=F(echo rgb )*weather_factor

[0021] wea=randomshift((dis+echo) / 2)

[0022] In the formula, d rgb is the distance mapped to 0-255 gray value; d is the distance, d max is the preset maximum distance, d min is the nearest distance contained in the data set, echo rgb is the gray value of the echo intensity channel generated according to the distance information; F operation means drawing a small circle according to the radar target position, gray value and radius information, and generating a corresponding single channel image; weather_factor is the weather factor; dis is the distance information channel, echo is the echo intensity channel; wea is the mixed channel, which adds the distance and echo intensity and then divides by 2, and applies randomshift random disturbance.

[0023] In the technical solution, step 2 is specifically: connecting the SimAM module for inferring 3D attention weights for each neuron to the rear of the shallow feature extraction module and the front of the feature fusion module.

[0024] In the technical solution, step 3 is specifically:

[0025] When the radar channel data stream passes through the first residual block of layer1 of resnet50, the shallow feature vector is stacked in front of the visible light channel feature vector of the same layer, and a 1*1 convolution operation is performed to reduce the channel number, so that it returns to the three-dimensional shape before stacking;

[0026] After the SimAM module, the corresponding bits are multiplied to end the fusion operation.

[0027] In the technical solution, step 3 is:

[0028] If the input information contains visible light, the visible light path is used as the main path; if it does not contain visible light, the infrared path is used as the main path, and the shallow feature layer of the remaining channel information is additionally added to the main path.

[0029] In the technical solution, step 4 specifically includes:

[0030] Step s41: According to the experimental requirements, meteorological conditions and input data sources are selected respectively, training and test sets are divided according to the rules, and data to be trained and tested is selected in the program interface;

[0031] Step s42: Set the model initialization parameters;

[0032] Epoch is selected as 80, batch_size is set to 32, input image size is set to 512*512, backbone is selected as resnet50, and whether to use a pre-trained model is selected according to the model convergence; The iou threshold is set to 0.25, the confidence threshold is set to 0.3; The initial learning rate is 0.0016, the minimum learning rate is 0.00001, and the cosine decay is used;

[0033] Step s43: Iterative training;

[0034] Use the Adam optimizer to optimize the parameters, save the model parameters after each complete training of an epoch, and test all models after training is completed, and save the test parameters.

[0035] In the technical solution, in step s43, the evaluation criteria include: AP, precision and recall.

[0036] In the technical solution, step 5 is specifically:

[0037] The test data in the test set is input into the trained network model, detection extraction is carried out using the improved feature extraction module, multi-scale feature fusion is carried out, then target detection is carried out using the Retinanet detection head; classification, confidence and position information of the target are obtained.

[0038] The present application has the following beneficial effects:

[0039] The SAF-FCOS-based multi-weather sea surface floating ice detection algorithm based on heterogenous data fusion can use registered data from different sensors under different weather conditions for sea surface floating ice detection based on the anchor-free SAF-FCOS.

[0040] The SAF-FCOS-based multi-weather sea surface floating ice detection algorithm based on heterogenous data fusion can select the required model according to the actual situation, which improves the overall robustness of the project.

[0041] The SAF-FCOS-based multi-weather sea surface floating ice detection algorithm based on heterogenous data fusion has high detection accuracy and speed in detecting sea ice under bad weather conditions, and can assist in sea surface floating ice target detection under various working conditions. BRIEF DESCRIPTION OF DRAWINGS

[0042] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0043] Figure 1 The present application is a basic flowchart of the SAF-FCOS-based multi-weather sea surface floating ice detection algorithm based on heterogenous data fusion.

[0044] Figure 2 It is a schematic diagram of a multi-channel network structure.

[0045] Figure 3 It is a schematic diagram of the 3d attention weight idea of the SimAM module.

[0046] Figure 4 It is a schematic diagram of the feature extraction part network structure after importing the SimAM module.

[0047] Figure 5 It is a schematic diagram of a hierarchical feature fusion network structure.

[0048] Figure 6 It is a schematic diagram of the final synthesis result of the radar message.

[0049] Figure 7 It is a schematic diagram of the effect of infrared and visible dual-channel sea ice detection in heavy fog (bad weather).

[0050] Figure 8Another infrared and visible dual-channel sea ice detection effect diagram in heavy fog (bad weather).

[0051] Figure 9 The detection results under several different fusion strategies are shown in the diagram. The three different color bar charts are respectively based on pixel-level fusion DCA, pyramid, and the multi-channel model used in the application. DETAILED DESCRIPTION

[0052] The multi-weather sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the application, as shown in the figure, includes the following steps: Figure 1

[0053] Step 1: Use heterogeneous data to simulate a data set, and divide the data set according to task requirements.

[0054] According to the task requirements, three different data sources are divided, and nine kinds of weather are divided. They include: simulated sea surface floating ice images in sunny, six kinds of fog, rainy and snowy weather, infrared images generated by grayscale, down-sampling, color inversion operation, and infrared images in different weather conditions generated by a generative adversarial network.

[0055] The specific steps are as follows:

[0056] Use existing floating ice images to simulate images of various sensors or radar message information under different weather conditions. These information is used for data fusion and detection;

[0057] When processing radar message information, the distance, echo intensity and weather condition in the radar data are regarded as features and mapped to the image coordinate system according to certain rules, and input into the network for training; the data set is divided into training set t1 and test set t2 according to certain rules.

[0058] The radar R channel is formed by echo intensity mapping. In this mapping method, it is considered that the echo intensity is inversely proportional to the square of the distance d. The G channel is generated by distance d mapping. The B channel is generated by random offset of the G channel data; the radar data mapping rule calculation formula is:

[0059]

[0060] echo=F(echo rgb )*weather_factor

[0061] wea=randomshift((dis+echo) / 2)

[0062] In order to facilitate program processing, the radar data sensitivity is increased, and a small circle is drawn at the corresponding position of the target. The radius of the circle is linearly mapped to the range of 10-20 by the d line of the radar message; in the formula, d​rgb is the distance mapped to 0-255 grayscale value; d is the distance, d max is the preset maximum distance 30km, d min is the nearest distance 1.4km contained in the data set, echo rgb is the grayscale value of the echo intensity channel generated according to the distance information, which is inversely proportional to the square of the distance; the F operation refers to drawing a small circle according to the radar marked target position, grayscale value, radius and other information, and generating a corresponding single-channel image; the weather_factor is a weather factor, different weather sets different weights, simulating the interference of weather on echo intensity; dis is the distance information channel, echo is the echo intensity channel; wea is a mixed channel, which adds the distance and echo intensity and then divides by 2, and applies randomshift random disturbance, which is used to represent the influence of distance and weather and the uncertainty of radar echo on target marking. The final synthesis result of the radar message is as shown in Figure 6 .

[0063] Step 2: Add attention mechanism module.

[0064] The feature extraction idea in the prior art is to perform convolution on all pixels of an image, and the feature extraction is relatively balanced, which increases the complexity of the extraction process and lacks targeted extraction of effective features.

[0065] The SimAM module is used to learn the part of the image to be processed, thereby reducing the number of pixels to be processed, reducing the complexity of the task, and improving the network efficiency.

[0066] The specific steps are as follows:

[0067] SimAM (Simple Attention Module) proposes a conceptually simple but very effective attention module for convolutional neural networks; unlike existing channel dimension and spatial dimension attention modules, SimAM can infer 3D attention weights for each neuron in the feature map without adding parameters to the original network; it uses a parameter-free design without adding any additional parameters, and can directly generate 3D weights, and the comprehensive attention mechanism can more accurately capture important feature information, as shown in Figure 2 and 3 .

[0068] The method of the present application connects the SimAM module behind the shallow feature extraction module and in front of the feature fusion module, as shown in Figure 4 .

[0069] Step 3: Feature fusion.

[0070] Different source feature vectors are stacked by using a fusion layer in the feature fusion module, and a convolution module in the feature fusion module is used to generate a feature vector of a specific channel number, so as to facilitate subsequent pixel-by-pixel multiplication fusion.

[0071] The specific steps are as follows:

[0072] When the radar channel data stream passes through the first residual block of layer1 of resnet50, the shallow layer feature vector is stacked in front of the visible light channel feature vector of the same layer, and a 1x1 convolution operation is performed to reduce the channel number, so that the three-dimensional shape before stacking is restored; after the SimAM module, the corresponding bit is multiplied and the fusion operation is completed.

[0073] The SAF-FCOS multi-meteorological sea surface floating ice detection algorithm based on the SAF-FCOS heterogeneous data fusion of the application improves the feature fusion method in the original SAF-FCOS. In the original method, only the feature vectors are simply multiplied, and the effective information of the feature maps at different scales cannot be effectively utilized. The SAF-FCOS multi-meteorological sea surface floating ice detection algorithm based on the SAF-FCOS heterogeneous data fusion of the application replaces the original direct multiplication with hierarchical fusion, and mainly uses the visible light path and the infrared path containing texture information in the fusion process, and uses other paths as auxiliary, thereby improving the target detection accuracy of sea ice.

[0074] In the prior art, the fusion method in the feature fusion is weighted addition or direct multiplication, which reduces the efficiency of feature fusion and causes waste of feature vector information.

[0075] In the application, if visible light is included in the input information, the visible light path is used as the main path; if no visible light is included, the infrared path is used as the main path, and the shallow layer feature layer of the remaining channel information is additionally added to the main path.

[0076] As shown in Figure 5 The feature fusion module mainly includes two levels, the channel number of the feature map is 256, and the size of the feature map is 128x128; the feature channel fusion layer and the 1x1 convolution module for reducing the channel number are included in the level; the fusion layer is responsible for stacking feature vectors of different sources, and the convolution module is responsible for generating a feature vector of a specific channel number, so as to facilitate subsequent pixel-by-pixel multiplication fusion; this is beneficial to reduce the operation amount of the network; after the fusion is completed, the channel number and size of the output feature map are 256 and 128x128; the improved multi-scale feature fusion module can more effectively utilize the feature information at different scales.

[0077] Step 4: model training.

[0078] The data of different data sources and meteorological conditions are trained respectively to obtain the optimal model suitable for various conditions.

[0079] The specific steps are:

[0080] Step s41: meteorological conditions and input data sources are selected according to experimental requirements, training and test sets are divided according to rules, and data to be trained and tested are selected in a program interface.

[0081] Step s42: set model initialization parameters; use two 4090 graphics cards for calculation, select 80 epochs, set batch_size to 32, set input image size to 512*512, select resnet50 as the backbone, and select whether to use a pre-trained model according to the model convergence; the iou threshold is set to 0.25, and the confidence threshold is set to 0.3; the initial learning rate is 0.0016, the minimum learning rate is 0.00001, and the cosine decay is used; if gradient anomaly occurs, the initial learning rate can be appropriately reduced.

[0082] Step s43: iterative training, using Adam optimizer for parameter optimization, saving model parameters after each complete training of an epoch, testing on all models after training is completed, and saving test parameters; the evaluation standard is selected as the evaluation standard in the target detection field, including AP, precision and recall.

[0083] The SAF-FCOS heterogenous data fusion based multi-weather sea surface floating ice detection algorithm of the application can select the required model according to the actual situation, which improves the robustness of the whole project.

[0084] Step 5: using the trained network model to detect the target of the test set of sea ice, obtaining the attribute, confidence and position information of the target and marking.

[0085] The specific steps are:

[0086] The test data in the test set is input into the trained network model, the improved feature extraction module is used for detection extraction and multi-scale feature fusion, and then the Retinanet detection head is used for target detection; the classification, confidence and position information of the target are obtained.

[0087] The SAF-FCOS heterogenous data fusion based multi-weather sea surface floating ice detection algorithm of the application will be described below with the training and detection examples of visible light and infrared dual channels, combined with the drawings.

[0088] The SAF-FCOS heterogenous data fusion based multi-weather sea surface floating ice detection algorithm of the application comprises the following steps:

[0089] Step 1: use heterogenous data to simulate a data set, and divide the data set according to task requirements.

[0090] The simulation generates sea surface ice maps under sunny, 6 kinds of fog, rainy and snowy weather, generates infrared images through grayscale, up and down sampling, color inversion and other operations, and generates infrared images under different weather conditions through the generative adversarial network.

[0091] Due to the image splicing method used to generate the dataset, the problem of target object repetition inevitably occurs. To avoid data set pollution, manually set the images numbered 1-10 as the test set, and the remaining valid images as the training set. The training set of sunny weather has 428 images, and the test set has 65 images. The training set of the remaining 8 kinds of weather has 247 images, and the test set has 38 images.

[0092] Step 2: Add attention mechanism module.

[0093] Build a dual-channel network model and import the SimAM module. The 3d weight of SimAM is as shown in Figure 3 . Adding the attention module to the rear of each channel feature extraction module can speed up model convergence, and the network structure is as shown in Figure 2 .

[0094] Step 3: Feature fusion.

[0095] During feature fusion, when the radar channel data stream passes through the first residual block of layer1 of resnet50, the shallow feature vector is stacked in front of the visible light channel feature vector of the same layer, and a 1x1 convolution operation is performed to reduce the channel number, so that it returns to the three-dimensional shape before stacking. After passing through the SimAM module, multiply the corresponding bits and end the fusion operation.

[0096] Step 4: Model training.

[0097] Set the hyperparameters and train the model, specifically:

[0098] Use 2 blocks of 4090 graphics cards for calculation, select 80 for epoch, set 32 for batch_size, set 512x512 for input image size, select resnet50 for backbone, and select whether to use a pre-trained model according to the model convergence.

[0099] Set the iou threshold to 0.25 and the confidence threshold to 0.3.

[0100] The initial learning rate is 0.0016, the minimum learning rate is 0.00001, and the cosine decay is used. If gradient anomalies occur, the initial learning rate can be appropriately reduced.

[0101] Iterative training is performed, the Adam optimizer is used for parameter optimization, the model parameters are saved after each complete training of an epoch, and the test parameters are saved after the training is completed.

[0102] The evaluation criteria are selected according to the target detection field, and the indicators used in the experiment include AP, precision and recall rate.

[0103] Step 5: The trained network model is used for target detection of sea ice in the test set, and the attribute, confidence and position information of the target are obtained and labeled.

[0104] The test data in the test set are input into the trained network model, the improved feature extraction module is used for detection extraction and multi-scale feature fusion, and then the Retinanet detection head is used for target detection; the classification, confidence and position information of the target are obtained.

[0105] Using the improved SAF-FCOS model of the application, after a given image, the trained model is used to detect the related information of sea ice under different weather conditions. Figure 7 and 8 The recall rate indexes of the detection of several different data fusion methods are shown in Figure 9 .

[0106] The multi-weather sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the application can use the registered data from different sensors under different weather conditions for sea surface ice detection based on the anchor-free SAF-FCOS.

[0107] The application of the multi-weather sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the application has high detection accuracy and speed in detecting sea ice under bad weather conditions, and can assist in detecting sea surface ice targets under various working conditions.

[0108] Obviously, the above embodiments are only examples for clear illustration, and are not limited to the embodiments. For ordinary skilled persons in the art, other different forms of changes or variations can be made on the basis of the above description. It is not necessary or possible to exhaust all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the application.

Claims

1. A multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion, characterized in that, Includes the following steps: Step 1: Use heterogeneous data to simulate and generate a dataset, and divide the dataset according to task requirements; Step 2: Add the attention mechanism module; Step 3: Feature fusion; The feature fusion module stacks feature vectors from different sources using a fusion layer, and generates feature vectors with a specific number of channels using a convolution module in the feature fusion module, so that subsequent pixel-by-pixel multiplication and fusion can be performed. Step 4: Model training; The model is trained on data from different data sources and meteorological conditions to obtain the optimal model that is suitable for various situations. Step 5: Use the trained network model to perform target detection on the sea ice in the test set, obtain the target's attributes, confidence level, and location information, and then label them; Step 1 is as follows: Use ice floe images to simulate and generate images or radar message information from various sensors under different meteorological conditions; The distance, echo intensity, and meteorological conditions in the radar message information are used as features and mapped to the image coordinate system according to certain rules, and then input into the network for training. In step 1, the radar message data is mapped to a series of small circles, and the calculation formula is as follows: echo=F(echo rgb )*weather-factor wea=randomshift((dis+echo) / 2) In the formula, d rgb This maps distance to grayscale values ​​between 0 and 255; d represents the distance. max To preset the maximum distance, d min echo is the nearest distance contained in the dataset. rgb The grayscale value of the echo intensity channel is generated based on the distance information mapping; the F operation refers to drawing a small circle based on the radar mark target position, grayscale value, and radius information to generate the corresponding single-channel image; weather_factor is the weather factor; dis is the distance information channel, echo is the echo intensity channel; wea is the mixed channel, which sums the distance and echo intensity and divides by 2, and applies random shift random perturbation.

2. The multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1, characterized in that, Step 2 specifically involves connecting the SimAM module, which infers 3D attention weights for each neuron, to the back of the shallow feature extraction module and the front of the feature fusion module.

3. The multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1, characterized in that, Step 3 specifically involves: When the radar channel data stream passes through the first residual block of layer 1 of ResNet50, its shallow feature vector is stacked in front of the feature vector of the same layer in the visible light channel, and then subjected to a 1×1 convolution operation that reduces the number of channels to restore it to its original three-dimensional shape before stacking. After passing through the SimAM module, the corresponding bits are multiplied, and the fusion operation ends.

4. The multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1, characterized in that, In step 3: If the input information contains visible light, the visible light path is used as the main path; if it does not contain visible light, the infrared path is used as the main path, and the shallow feature layer of the remaining channel information is added to the main path.

5. The multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1, characterized in that, Step 4 specifically includes: Step s41: Select meteorological conditions and input data sources according to the experimental requirements, divide the training and test sets according to the rules, and select the data to be trained and tested in the program interface; Step s42: Set model initialization parameters; The epoch is set to 80, the batch size to 32, the input image size to 512×512, the backbone to ResNet50, and whether to use a pre-trained model is selected based on the model's convergence. The IoU threshold is set to 0.25, the confidence threshold is set to 0.3, the initial learning rate is 0.0016, and the minimum learning rate is 0.00001 cosine decay. Step s43: Iterative training; The Adam optimizer is used for parameter optimization. The model parameters are saved after each complete training epoch. After training is completed, the model is tested on all models and the test parameters are saved.

6. The multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 5, characterized in that, In step s43, the evaluation criteria include: AP, precision, and recall.

7. The multi-meteorological sea surface ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1, characterized in that, Step 5 specifically involves: Test data from the test set is input into the trained network model, and the improved feature extraction module is used to extract detection data and perform multi-scale feature fusion. Then, the RetinaNet detection head is used for target detection. We obtain the target's classification, confidence level, and location information.

Images