Multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogenous data fusion

Abstract

The invention relates to a multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion, and relates to the technical field of computer vision artificial intelligence target detection. The multi-meteorological sea surface floating ice detection algorithm comprises the following steps: simulating to generate a data set by using heterogenous data, and dividing the data set according to task requirements; adding an attention mechanism module; feature fusion; model training; and performing target detection on the sea ice of the test set by using the trained network model. According to the multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion, registered data from different sensors under different meteorological conditions can be used for sea surface floating ice detection based on the anchor-free of SAF-FCOS; the required model can be selected according to the actual situation, and the overall robustness of the project is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision artificial intelligence target detection, and in particular to a multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion. Background Art

[0002] Sea surface ice detection plays an important role in climate research, maritime safety, and disaster monitoring and early warning. The state of sea ice is a basic parameter that describes the polar environment and is of great significance to climate change research. By monitoring sea ice changes, we can improve our understanding of the air-ice-sea interaction mechanism and support climate research and forecasting work; sea ice detection also provides important information support for coastal governments, offshore operations, etc., which helps to ensure maritime safety and respond to potential threats from sea ice in a timely manner; through remote sensing monitoring and other technical means, we can grasp the type and distribution of sea ice in real time, improve the monitoring and early warning capabilities of marine disasters, and reduce disaster losses.

[0003] Deep learning has been widely used in the field of image target detection, especially the SAF-FCOS network, which has excellent performance in single-stage networks without anchor frames. With the powerful self-learning ability of deep learning, the problem of low efficiency of traditional object detection can be solved. However, the original SAF-FCOS network is for multi-category target detection, so it is not fully suitable for sea ice detection under multiple weather conditions. When the meteorological conditions are slightly complicated, the recall rate and accuracy of the detection effect under a single channel are significantly reduced; the feature fusion module of the original network is only a simple multiplication of the pixel values ​​at the corresponding position. When detecting sea ice targets, due to the small scale, it is impossible to use multi-layer feature information, and the representation ability is low, which will cause some false detection and missed detection. Summary of the invention

[0004] The present invention aims to solve the technical problems in the prior art and provides a multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] A multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion includes the following steps:

[0007] Step 1: Use heterogeneous data to simulate and generate a data set, and divide the data set according to task requirements;

[0008] Step 2: Add the attention mechanism module;

[0009] Step 3: Feature fusion;

[0010] The fusion layer in the feature fusion module is used to stack feature vectors from different sources, and the convolution module in the feature fusion module is used to generate feature vectors with a specific number of channels so that subsequent pixel-by-pixel multiplication and fusion can be performed.

[0011] Step 4: Model training;

[0012] Train data from different data sources and meteorological conditions separately to obtain the optimal model suitable for various situations;

[0013] Step 5: Use the trained network model to detect the sea ice in the test set, obtain the target’s attributes, confidence, and location information, and mark them.

[0014] In the above technical solution, step 1 is specifically as follows:

[0015] Use floating ice image simulation to generate images or radar message information of various sensors under different meteorological conditions;

[0016] The distance, echo intensity and meteorological conditions in the radar message information are used as features and mapped to the image coordinate system according to certain rules and input into the network for training.

[0017] In the above technical solution, the radar message data in step 1 is mapped into a series of small circles, and the calculation formula is:

[0018]

[0019]

[0020] dis=F(d rgb )echo=F(echo rgb )*weather_factor

[0021] wea=randomshift((dis+echo) / 2)

[0022] Where, d rgb The distance is mapped to a grayscale value of 0-255; d is the distance, d max is the preset maximum distance, d min The closest distance contained in the data set, echo rgb is the gray value of the echo intensity channel generated based on the distance information mapping; the F operation refers to drawing a small circle according to the radar target position, gray value, and radius information to generate the corresponding single-channel image; weather_factor is the weather factor; dis is the distance information channel, echo is the echo intensity channel; wea is a mixed channel, which adds the distance and echo intensity and divides it by 2, and applies randomshift random perturbation.

[0023] In the above technical solution, step 2 is specifically: connecting the SimAM module that infers the 3D attention weight for each neuron to the back of the shallow feature extraction module and the front of the feature fusion module.

[0024] In the above technical solution, step 3 is specifically as follows:

[0025] When the radar channel data stream passes through the first residual block of layer 1 of resnet50, its shallow feature vector is stacked in front of the feature vector of the same layer of the visible light channel, and then undergoes a 1×1 convolution operation with a reduced number of channels to restore it to its three-dimensional shape before stacking;

[0026] After the corresponding bits are multiplied after passing through the SimAM module, the fusion operation ends.

[0027] In the above technical solution, in step 3:

[0028] If the input information contains visible light, the visible light path is used as the main path; if it does not contain visible light, the infrared path is used as the main path, and the shallow feature layer of the remaining channel information is additionally added to the main path.

[0029] In the above technical solution, step 4 specifically includes:

[0030] Step s41: Select meteorological conditions and input data sources according to the experimental requirements, divide the training and test sets according to the rules, and select the data to be trained and tested in the program interface;

[0031] Step s42: setting model initialization parameters;

[0032] The epoch is set to 80, the batch_size is set to 32, the input image size is set to 512×512, the backbone is selected as resnet50, and whether to use the pre-trained model is selected based on the model convergence; the iou threshold is set to 0.25, and the confidence threshold is set to 0.3; the initial learning rate is 0.0016, and the minimum learning rate is 0.00001 cosine decay;

[0033] Step s43: iterative training;

[0034] Use the Adam optimizer for parameter optimization, save the model parameters after each complete training epoch, test all models after training is completed, and save the test parameters.

[0035] In the above technical solution, in step s43, the evaluation criteria include: AP, precision and recall.

[0036] In the above technical solution, step 5 is specifically as follows:

[0037] The test data in the test set is input into the trained network model, and the improved feature extraction module is used for detection extraction and multi-scale feature fusion. Then, the Retinanet detection head is used for target detection; the classification, confidence and location information of the target are obtained.

[0038] The present invention has the following beneficial effects:

[0039] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention is anchor-free based on SAF-FCOS and can use the registered data from different sensors under different meteorological conditions for the detection of sea surface floating ice.

[0040] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention can select the required model according to the actual situation, which improves the overall robustness of the project.

[0041] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention can detect sea ice under severe meteorological conditions with high detection accuracy and compliance speed, and can assist in the detection of sea surface floating ice targets under various working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0043] Figure 1 It is a basic flow chart of the multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention.

[0044] Figure 2 Schematic diagram of multi-channel network structure.

[0045] Figure 3 Schematic diagram of the 3D attention weight concept of the SimAM module.

[0046] Figure 4 This is a schematic diagram of the network structure of the feature extraction part after importing the SimAM module.

[0047] Figure 5 Schematic diagram of the hierarchical feature fusion network structure.

[0048] Figure 6 The figure is a schematic diagram of the synthesized RGB radar, showing the final synthesis result of the radar message.

[0049] Figure 7 Schematic diagram of the effect of infrared and visible dual-channel sea ice detection on foggy days (bad weather).

[0050] Figure 8This is another schematic diagram of the effect of infrared and visible dual-channel sea ice detection on a foggy day (bad weather).

[0051] Fig. 9 Schematic diagrams of detection results under several different fusion strategies. Among them, the three bar graphs with different colors are schematic diagrams of DCA based on pixel-level fusion, pyramid and multi-channel model used in the present invention. DETAILED DESCRIPTION

[0052] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0053] Step 1: Use heterogeneous data to simulate and generate a dataset, and divide the dataset according to task requirements.

[0054] According to the task requirements, three different data sources are divided into 9 types of weather, including: simulating the generation of sea ice images on sunny days, 6 types of foggy days, rainy days and snowy days, generating infrared images through grayscale, up and down sampling, and color inversion operations, and generating infrared images under different meteorological conditions through generative adversarial networks.

[0055] The specific steps are:

[0056] Use existing floating ice images to simulate and generate images or radar message information of various sensors under different meteorological conditions, which are used for data fusion and detection;

[0057] When processing radar message information, the distance, echo intensity and meteorological conditions in the radar data are used as features and mapped to the image coordinate system according to certain rules, and then input into the network for training; the data set is divided into training set t1 and test set t2 according to certain rules.

[0058] The radar R channel is formed by echo intensity mapping. In this mapping method, the echo intensity is considered to be inversely proportional to the square of the distance d. The G channel is generated by distance d mapping, and the B channel is generated by random offset of the G channel data. The radar data mapping rule calculation formula is:

[0059]

[0060] echo=F(echo rgb )*weather_factor

[0061] wea=randomshift((dis+echo) / 2)

[0062] In order to facilitate program processing and increase the sensitivity of radar data, a small circle is drawn at the corresponding position of the target. The radius of the circle is linearly mapped from the d of the radar message to the range of 10 to 20; where drgb The distance is mapped to a grayscale value of 0-255; d is the distance, d max The preset maximum distance is 30km, d min The closest distance in the data set is 1.4 km, echo rgb It is the gray value of the echo intensity channel generated based on the distance information mapping, which is inversely proportional to the square of the distance; the F operation refers to drawing a small circle based on the radar target position, gray value, radius and other information to generate the corresponding single-channel image; weathor_factor is the weather factor, and different weights are set for different weather conditions to simulate the interference of weather on echo intensity. dis is the distance information channel, echo is the echo intensity channel; wea is a mixed channel, which adds the distance and echo intensity and divides it by 2, and applies randomshift random perturbation to characterize the influence of distance and weather and the uncertainty of radar echo on target marking. The final synthesis result of the radar message is as follows: Figure 6 shown.

[0063] Step 2: Add the attention mechanism module.

[0064] The idea of ​​feature extraction in the prior art is to perform convolution on all pixels of an image and extract features relatively evenly, which increases the complexity of the extraction process and lacks targeted extraction of effective features.

[0065] The present invention uses the SimAM module to learn the part of the image to be processed, thereby reducing the number of pixels that need to be processed, reducing the complexity of the task, and improving network efficiency.

[0066] The specific steps are:

[0067] SimAM (Simple Attention Module) proposes a conceptually simple but very effective attention module for convolutional neural networks. Unlike existing channel-dimensional and spatial-dimensional attention modules, SimAM can infer 3D attention weights for each neuron in the feature map without adding parameters to the original network. It adopts a parameter-free design without adding any additional parameters and can directly generate 3D weights. The comprehensive attention mechanism can more accurately capture important feature information, such as Figure 2 and 3 shown.

[0068] The method of the present invention connects the SimAM module to the back of the shallow feature extraction module and the front of the feature fusion module, such as Figure 4 shown.

[0069] Step 3: Feature fusion.

[0070] The fusion layer in the feature fusion module is used to stack feature vectors from different sources, and the convolution module in the feature fusion module is used to generate feature vectors with a specific number of channels so that subsequent pixel-by-pixel multiplication and fusion can be performed.

[0071] The specific steps are:

[0072] When the radar channel data stream passes through the first residual block of layer1 of resnet50, its shallow feature vector is stacked in front of the feature vector of the same layer of the visible light channel, and undergoes a 1×1 convolution operation to reduce the number of channels to restore it to the three-dimensional shape before stacking; after passing through the SimAM module, the corresponding bits are multiplied and the fusion operation is ended.

[0073] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention improves the feature fusion method in the original SAF-FCOS. The original method simply multiplies the feature vectors, and cannot effectively utilize the effective information of feature maps at different scales. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention replaces the original direct multiplication with layered and step-by-step fusion, and in the fusion process, the visible light path and infrared path containing texture information are mainly used, and other paths are used as auxiliary, thereby improving the target detection accuracy of sea ice.

[0074] When performing feature fusion, the fusion methods in the prior art are all weighted addition or direct multiplication, which will reduce the efficiency of feature fusion and cause information waste of feature vectors.

[0075] In the present invention, if the input information contains visible light, the visible light path is used as the main path; if it does not contain visible light, the infrared path is used as the main path, and the shallow feature layer of the remaining channel information is additionally added to the main path.

[0076] like Figure 5 As shown in the figure, the feature fusion module mainly includes two layers, the number of feature map channels is 256, and the feature map size is 128×128; the layer includes a feature channel fusion layer and a 1×1 convolution module with reduced channel number; the fusion layer is responsible for stacking feature vectors from different sources, and the convolution module is responsible for generating feature vectors with a specific number of channels, which is convenient for subsequent pixel-by-pixel multiplication and fusion; this is conducive to reducing the amount of network calculations; after the fusion is completed, the number and size of the output feature map channels are 256 and 128×128; the improved multi-scale feature fusion module can more effectively utilize feature information at different scales.

[0077] Step 4: Model training.

[0078] Data from different data sources and meteorological conditions are trained separately to obtain the optimal model suitable for various situations.

[0079] The specific steps are:

[0080] Step s41: Select meteorological conditions and input data sources according to the experimental requirements, divide the training and test sets according to the rules, and select the data to be trained and tested in the program interface.

[0081] Step s42: Set the model initialization parameters; use two 4090 graphics cards for calculation, select 80 epochs, batch_size to 32, input image size to 512×512, backbone to resnet50, and choose whether to use the pre-trained model based on the model convergence; the iou threshold is set to 0.25, and the confidence threshold is set to 0.3; the initial learning rate is 0.0016, and the minimum learning rate is 0.00001 cosine decay; if the gradient is abnormal, the initial learning rate can be appropriately reduced.

[0082] Step s43: Iterate training and use the Adam optimizer to optimize parameters. Save the model parameters after each complete training epoch. After training, test all models and save the test parameters. The evaluation criteria are selected from the target detection field, including AP, precision, and recall.

[0083] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention can select the required model according to the actual situation, which improves the overall robustness of the project.

[0084] Step 5: Use the trained network model to detect the sea ice in the test set, obtain the target’s attributes, confidence, and location information, and mark them.

[0085] The specific steps are:

[0086] The test data in the test set is input into the trained network model, and the improved feature extraction module is used for detection extraction and multi-scale feature fusion. Then, the Retinanet detection head is used for target detection; the classification, confidence and location information of the target are obtained.

[0087] The following is an example of training and detection of visible light and infrared dual channels, combined with the accompanying drawings to illustrate the multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention.

[0088] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention comprises the following steps:

[0089] Step 1: Use heterogeneous data to simulate and generate a dataset, and divide the dataset according to task requirements.

[0090] The simulation generates images of floating ice on the sea surface on sunny days, six types of foggy days, rainy days and snowy days. Infrared images are generated through grayscale conversion, up and down sampling, color inversion and other operations. Infrared images under different meteorological conditions are simulated through generative adversarial networks.

[0091] Since the dataset is generated by image stitching, the problem of repeated target objects is inevitable. To avoid dataset contamination, the images with target numbers 1 to 10 are manually set as the test set, and the remaining valid images are set as the training set. There are 428 training sets and 65 test sets for sunny days, and 247 training sets and 38 test sets for the other 8 weather conditions.

[0092] Step 2: Add the attention mechanism module.

[0093] Build a dual-channel network model and import it into the SimAM module. The 3D weights of SimAM are as follows Figure 3 As shown in Figure 2. Adding the attention module after each channel feature extraction module can speed up the model convergence. The network structure is shown in Figure 2. Figure 2 shown.

[0094] Step 3: Feature fusion.

[0095] During feature fusion, when the radar channel data stream passes through the first residual block of layer1 of resnet50, its shallow feature vector is stacked in front of the feature vector of the same layer of the visible light channel, and a 1×1 convolution operation with a reduced number of channels is performed to restore it to the three-dimensional shape before stacking. After passing through the SimAM module, the corresponding bits are multiplied and the fusion operation is terminated.

[0096] Step 4: Model training.

[0097] Set hyperparameters and train the model, specifically:

[0098] Use two 4090 graphics cards for calculation, select 80 epochs, set batch_size to 32, set the input image size to 512×512, select resnet50 as the backbone, and choose whether to use the pre-trained model based on the model convergence.

[0099] The iou threshold is set to 0.25 and the confidence threshold is set to 0.3.

[0100] The initial learning rate is 0.0016, and the minimum learning rate is 0.00001 cosine decay. If the gradient is abnormal, the initial learning rate can be appropriately reduced.

[0101] Perform iterative training and use the Adam optimizer for parameter optimization. Save the model parameters after each complete training epoch. After training, test all models and save the test parameters.

[0102] Evaluation criteria are selected in the field of target detection. The indicators used in the experiment include: AP, precision and recall.

[0103] Step 5: Use the trained network model to detect the sea ice in the test set, obtain the target’s attributes, confidence, and location information, and mark them.

[0104] The test data in the test set is input into the trained network model, and the improved feature extraction module is used for detection extraction and multi-scale feature fusion. Then, the Retinanet detection head is used for target detection; the classification, confidence and location information of the target are obtained.

[0105] Using the improved SAF-FCOS model of the present invention, after a given image, the trained model is used to detect information related to sea ice under different weather conditions. The detection results are as follows: Figure 7 and 8 As shown; the recall rate indicators of several different data fusion methods are as follows Fig. 9 shown.

[0106] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention is anchor-free based on SAF-FCOS and can use the registered data from different sensors under different meteorological conditions for the detection of sea surface floating ice.

[0107] The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion of the present invention can detect sea ice under severe meteorological conditions with high detection accuracy and compliance speed, and can assist in the detection of sea surface floating ice targets under various working conditions.

[0108] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion, characterized in that: The following steps are involved: Step 1: Use heterogeneous data to simulate and generate a data set, and divide the data set according to task requirements; Step 2: Add the attention mechanism module; Step 3: Feature fusion; The fusion layer in the feature fusion module is used to stack feature vectors from different sources, and the convolution module in the feature fusion module is used to generate feature vectors with a specific number of channels so that subsequent pixel-by-pixel multiplication and fusion can be performed. Step 4: Model training; Train the data from different data sources and meteorological conditions separately to obtain the optimal model suitable for various situations; Step 5: Use the trained network model to detect the sea ice in the test set, obtain the target’s attributes, confidence, and location information, and mark them.

2. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1 is characterized in that: Step 1 is as follows: Use floating ice image simulation to generate images or radar message information of various sensors under different meteorological conditions; The distance, echo intensity and meteorological conditions in the radar message information are used as features and mapped to the image coordinate system according to certain rules, and then input into the network for training.

3. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 2 is characterized in that: In step 1, the radar message data is mapped into a series of small circles, and the calculation formula is: echo=F(echo rgb )*weather-factor wea=randomshift((dis+echo) / 2) Where, d rgb The distance is mapped to a grayscale value of 0-255; d is the distance, d max is the preset maximum distance, d min The closest distance contained in the data set, echo rgb is the gray value of the echo intensity channel generated based on the distance information mapping; the F operation refers to drawing a small circle according to the radar target position, gray value, and radius information to generate the corresponding single-channel image; weather-factor is the weather factor; dis is the distance information channel, echo is the echo intensity channel; wea is a mixed channel, which adds the distance and echo intensity and divides it by 2, and applies randomshift random perturbation.

4. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1 is characterized in that: Step 2 is as follows: Connect the SimAM module that infers 3D attention weights for each neuron to the back of the shallow feature extraction module and the front of the feature fusion module.

5. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1 is characterized in that: Step 3 is as follows: When the radar channel data stream passes through the first residual block of layer 1 of resnet50, its shallow feature vector is stacked in front of the feature vector of the same layer of the visible light channel, and then undergoes a 1×1 convolution operation with a reduced number of channels to restore it to its three-dimensional shape before stacking; After the corresponding bits are multiplied after passing through the SimAM module, the fusion operation ends.

6. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1 is characterized in that: In step 3: If the input information contains visible light, the visible light path is used as the main path; if it does not contain visible light, the infrared path is used as the main path, and the shallow feature layer of the remaining channel information is additionally added to the main path.

7. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1 is characterized in that: Step 4 specifically includes: Step s41: Select meteorological conditions and input data sources according to the experimental requirements, divide the training and test sets according to the rules, and select the data to be trained and tested in the program interface; Step s42: setting model initialization parameters; The epoch is set to 80, the batch_size is set to 32, the input image size is set to 512×512, the backbone is selected as resnet50, and whether to use the pre-trained model is selected based on the model convergence; the iou threshold is set to 0.25, and the confidence threshold is set to 0.3; the initial learning rate is 0.0016, and the minimum learning rate is 0.00001 cosine decay; Step s43: iterative training; Use the Adam optimizer for parameter optimization, save the model parameters after each complete training epoch, test all models after training is completed, and save the test parameters.

8. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 7 is characterized in that: In step s43, the evaluation criteria include: AP, precision and recall.

9. The multi-meteorological sea surface floating ice detection algorithm based on SAF-FCOS heterogeneous data fusion according to claim 1 is characterized in that: Step 5 is as follows: The test data in the test set is input into the trained network model, and the improved feature extraction module is used for detection extraction and multi-scale feature fusion, and then the Retinanet detection head is used for target detection; Get the classification, confidence and location information of the target.

Images