Breeding area prediction method based on improved nested UNet neural network

Abstract

The invention discloses a cultivation area prediction method based on an improved nested UNet neural network, and belongs to the field of mariculture area prediction methods. According to the method, data preprocessing is carried out on high-resolution third radar image data and high-resolution second optical image data, wave band synthesis and image fusion are carried out on the optical image data and the radar image data, and a data set is constructed. On the basis of applying a U2-Net neural network model main body structure, a model network structure of an RSU module is modified, channel attention mechanisms based on maximum pooling and average pooling are added inside and outside a residual U-shaped module structure, a U2-Net SEOUT model is constructed, and prediction of the area and distribution of a breeding area is realized. According to the method, the advantages of the optical image and the SAR image are comprehensively utilized, the network structure is modified, the number of network layers and complexity are increased, and the prediction precision of the raft cultivation area is improved.

Description

Technical Field

[0001] The invention discloses a breeding area prediction method based on an improved nested UNet neural network, belonging to the field of breeding area prediction methods. Background Art

[0002] The existing prediction methods for raft aquaculture areas mainly include visual interpretation, feature index, deep learning and object-oriented, etc. Most of them are based on the extraction of a single data source by neural networks, and the prediction accuracy of aquaculture areas is limited. In addition, raft aquaculture facilities are often partially flooded, resulting in weak spectral information. Both traditional prediction methods and deep learning methods have certain limitations in the prediction accuracy of aquaculture areas. In recent years, with the rapid development of deep convolutional neural networks (CNNs), especially the rapid rise of fully convolutional networks (FCNs) in the field of regional prediction, the level of prediction of significant targets has achieved substantial progress.

[0003] The deep learning method used in the present invention has powerful data mining and feature extraction capabilities. It can autonomously learn the image spectral information and spatial features through the convolution layer, avoid complex spectral feature analysis of the image, reduce the workload, and simultaneously achieve large-scale automatic and accurate classification by fusing radar images and optical image data, and has the advantages of robustness, stability and invariance. SAR images are not affected by weather such as clouds and rain, and can be observed all day and all weather. The Gaofen-3 satellite image used in the present invention is one of the typical SAR satellite images. The present invention improves U by adding a channel attention mechanism. 2 -Net convolutional neural network, which comprehensively utilizes the advantages of optical images and SAR images, improves the accuracy and reliability of raft aquaculture area prediction. Summary of the invention

[0004] The purpose of the present invention is to provide a breeding area prediction method based on an improved nested UNet neural network, by increasing the number of network layers and complexity of the UNet network, increasing the scale of the feature map, and by comprehensively utilizing the advantages of optical images and SAR images, so as to improve the accuracy and reliability of raft breeding area prediction.

[0005] The breeding area prediction method based on the improved nested UNet neural network is characterized by comprising:

[0006] S1. Acquire Gaofen-3 image data of the breeding area through the Gaofen-3 satellite, and perform data preprocessing on the Gaofen-3 image data;

[0007] S2. Obtain radar image data of the breeding area by resampling and image registration processing the pre-processed Gaofen-3 image data;

[0008] S3. Obtaining Gaofen-2 image data of the breeding area through the Gaofen-2 satellite, performing data preprocessing on the Gaofen-2 image data, and obtaining optical image data of the breeding area;

[0009] S4. Construct the GF23B dataset by performing band synthesis and image fusion on the radar image data and optical image data of the breeding area;

[0010] S5. Based on the original UNet neural network model, the model complexity is improved by modifying the model network structure to obtain a nested UNet neural network;

[0011] S6. By adding a channel attention mechanism to the nested UNet neural network, U 2 -Net_Se 2 Neural network models;

[0012] S7. U 2 -Net_Se 2 The neural network model is trained and verified by U 2 -Net_Se 2 The neural network model realizes the prediction of breeding areas.

[0013] In S1, data preprocessing of GF-3 image data includes radiometric calibration, data conversion, multi-view processing, filtering processing and geocoding.

[0014] In S3, the data preprocessing of GF-2 image data includes radiometric calibration and atmospheric correction.

[0015] In S4, the radar image data of the breeding area is used as a band and the band is synthesized with the optical image data of the breeding area.

[0016] In S4, the image fusion algorithm NNDiffuse Pan Sharpening is used to perform pixel-level image fusion on the radar image data and the optical image data of the breeding area. The spatial and spectral features of the two image data are fused through image fusion, and the texture features of the image data are increased. The fused image data and the label data corresponding to the image data are uniformly cropped to a size of 512×512 to construct the GF23B dataset.

[0017] In S5, the nested UNet neural network is a two-layer nested U-shaped structure, including an encoder, a decoder and a saliency map fusion module. The encoder consists of six roadside unit RSU modules from En_1 to En_6, and the decoder consists of five roadside unit RSU modules from De_1 to De_5. The saliency map fusion module is connected to the encoder and the decoder, and feature fusion is performed in a gradually connected manner. The En_6 module of the encoder and the De_5 to De_1 modules of the decoder are subjected to 3×3 convolution and upsampling processing, and the output image is processed to be consistent with the size of the input image. After 1×1 convolution processing, the activation function Sigmoid function is used to generate a saliency probability map.

[0018] The RSU-L module is a highly determined RSU module, where L is the height, which is determined by the spatial resolution of the input feature. The height of the RSU module of the En_5 module is 7, and the height of the RSU module of the En_6 module is 4. Through pooling and upsampling operations, the RSU-7 module of the En_5 module and the RSU-4 module of the En_6 module are replaced with the RSU-4F module, where F represents extension. The resolution of the intermediate feature map is processed to be the same as the resolution of the input feature map through the RSU-4F module.

[0019] In S6, the U-shaped structure of the nested UNet neural network is constructed by adding channel attention modules both inside and outside. 2 -Net_Se 2 Model.

[0020] The channel attention module is a SE module based on global maximum pooling and average pooling, which is located inside and outside the structure of the En_1 to En_4 modules of the encoder part of the U-shaped structure of the nested UNet neural network.

[0021] In S7, the training set of U based on the GF23B dataset 2 -Net_Se 2 The neural network model is used for model training. The spatial and spectral characteristics of the image data in the GF23B dataset are learned by the model itself to predict the area and distribution of the breeding area. The model is verified based on the validation set of the GF23B dataset to test the prediction accuracy of the model.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The breeding area prediction method of the present invention is based on U 2 -Net neural network model, modify the model structure, replace the En_5-En_6 module of the encoder with the RSU-4F module of the extended convolution, and build the U-shaped module by adding a channel attention module based on maximum pooling and average pooling.2 -Net_Se 2 The model increases the number of network layers and complexity of the neural network, and improves the accuracy of the prediction of the breeding area. In addition, the present invention performs band synthesis and image fusion on the GF-2 optical image and the GF-3 SAR image, constructs the GF23B data set, and can comprehensively utilize the advantages of optical images and SAR images, solving the problem that the spectral information of raft breeding facilities is weak due to partial flooding, which affects the prediction accuracy of the breeding area.

[0024] To U 2 -Net neural network internal and external U-shaped structure while adding channel attention module to form U 2 -Net_Se2 model has excellent results in extracting raft aquaculture areas. Its overall extraction accuracy for typical areas reached 98.05%, and the Kappa coefficient was 0.9589, which was 4.32% and 0.095 higher than that of the original neural network, respectively. For weak signal areas, its extraction accuracy reached more than 96.50%, which was more than 8.5% higher than that of the original neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a breeding area prediction method based on an improved nested UNet neural network provided in an embodiment of the present invention;

[0026] Figure 2 A network structure diagram of a UNet neural network provided in an embodiment of the present invention;

[0027] Figure 3 A network structure diagram of a UNet neural network provided in an embodiment of the present invention after adding a channel attention module;

[0028] Figure 4 A structural diagram of an RSU module with a height L of 7 provided in an embodiment of the present invention;

[0029] Figure 5 A diagram of a channel attention module based on maximum pooling and average pooling provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The breeding area prediction method based on the improved nested UNet neural network is characterized by comprising:

[0032] S1. Acquire Gaofen-3 image data of the breeding area through the Gaofen-3 satellite, and perform data preprocessing on the Gaofen-3 image data;

[0033] S2. Obtain radar image data of the breeding area by resampling and image registration processing the pre-processed Gaofen-3 image data;

[0034] S3. Obtaining Gaofen-2 image data of the breeding area through the Gaofen-2 satellite, performing data preprocessing on the Gaofen-2 image data, and obtaining optical image data of the breeding area;

[0035] S4. Construct the GF23B dataset by performing band synthesis and image fusion on the radar image data and optical image data of the breeding area;

[0036] S5. Based on the original UNet neural network model, the model complexity is improved by modifying the model network structure to obtain a nested UNet neural network;

[0037] S6. By adding a channel attention mechanism to the nested UNet neural network, U 2 -Net_Se 2 Neural network models;

[0038] S7. U 2 -Net_Se 2 The neural network model is trained and verified by U 2 -Net_Se 2 The neural network model realizes the prediction of breeding areas.

[0039] In S1, data preprocessing of GF-3 image data includes radiometric calibration, data conversion, multi-view processing, filtering processing and geocoding.

[0040] In S3, the data preprocessing of GF-2 image data includes radiometric calibration and atmospheric correction.

[0041] In S4, the radar image data of the breeding area is used as a band and the band is synthesized with the optical image data of the breeding area.

[0042] In S4, the image fusion algorithm NNDiffuse Pan Sharpening is used to perform pixel-level image fusion on the radar image data and the optical image data of the breeding area. The spatial and spectral features of the two image data are fused through image fusion, and the texture features of the image data are increased. The fused image data and the label data corresponding to the image data are uniformly cropped to a size of 512×512 to construct the GF23B dataset.

[0043] In S5, the nested UNet neural network is a two-layer nested U-shaped structure, including an encoder, a decoder and a saliency map fusion module. The encoder consists of six roadside unit RSU modules from En_1 to En_6, and the decoder consists of five roadside unit RSU modules from De_1 to De_5. The saliency map fusion module is connected to the encoder and the decoder, and feature fusion is performed in a gradually connected manner. The En_6 module of the encoder and the De_5 to De_1 modules of the decoder are subjected to 3×3 convolution and upsampling processing, and the output image is processed to be consistent with the size of the input image. After 1×1 convolution processing, the activation function Sigmoid function is used to generate a saliency probability map.

[0044] The RSU-L module is a highly determined RSU module, where L is the height, which is determined by the spatial resolution of the input feature. The height of the RSU module of the En_5 module is 7, and the height of the RSU module of the En_6 module is 4. Through pooling and upsampling operations, the RSU-7 module of the En_5 module and the RSU-4 module of the En_6 module are replaced with the RSU-4F module, where F represents extension. The resolution of the intermediate feature map is processed to be the same as the resolution of the input feature map through the RSU-4F module.

[0045] In S6, the channel attention module is a SE module based on global maximum pooling and average pooling, located in the En_1 to En_4 module positions of the encoder part of the U-shaped structure of the nested UNet neural network. By adding the channel attention module both inside and outside the U-shaped structure of the nested UNet neural network, the U 2 -Net_Se 2 Model; By adding a channel attention module inside the structure of En_1 to En_4 modules, U 2 -Net_seIN model; U is constructed by adding a channel attention module outside the structure of En_1 to En_4 modules 2 -Net_SEOUT model. Based on GF23B dataset, U 2 -Net_seIN model, U 2 -Net_SEOUT model and U 2 -Net_Se 2 The model is used to compare and analyze the model accuracy and reliability, determine the optimal network model, and make breeding area predictions. 2 -Net_Se 2 , U 2 -Net_SEOUT,U 2 -Net_seIN's three improved models have better prediction effects on typical areas and weak signal areas than the original U 2-Net models are improved. The overall accuracy of typical area prediction is 4.32%, 2.19%, and 2.87% higher than that of the original model, and the Kappa coefficient is 0.0946, 0.0482, and 0.0631 higher, respectively. The overall accuracy of weak signal area extraction is 8.72%, 0.84%, and 3.16% higher than that of the original model, and the Kappa coefficient is 0.1758, 0.0171, and 0.0639 higher, respectively, which proves the effectiveness of the improved model.

[0046] In S7, the training set of U based on the GF23B dataset 2 -Net_Se 2 The neural network model is used for model training. The spatial and spectral features of the image data in the GF23B dataset are learned by the model to predict the area and distribution of the breeding area. The model is validated based on the validation set of the GF23B dataset to test the prediction accuracy of the model. 2 -Net_Se 2 The neural network model has excellent results in predicting raft aquaculture areas. The overall accuracy of the prediction for typical areas reached 98.05%, and the Kappa coefficient was 0.9589, which was 4.32% and 0.095 higher than that of the original neural network, respectively. For weak signal areas, the prediction accuracy reached more than 96.50%, which was more than 8.5% higher than that of the original neural network.

[0047] Embodiments of the present invention.

[0048] Figure 1 The flowchart of the breeding area prediction method based on the improved nested UNet neural network provided in the embodiment of the present invention first obtains the GF-3 image data, performs data preprocessing on the GF-3 image data, including radiation calibration, data conversion, multi-view processing, filtering processing and geocoding, and then resamples and image aligns the processed GF-3 image to obtain the radar image data of the breeding area. The GF-2 image data is obtained, and the GF-2 image data is subjected to radiation calibration and atmospheric correction to obtain the optical image data of the breeding area. The preprocessed GF-2 image and the registered GF-3 image are subjected to band synthesis and image fusion, and the GF23B data set is constructed based on the optical image data using the fused image data. Constructing UNet with channel attention mechanism added 2 -Net neural network models, including U 2 -Net_seIN model, U 2 -Net_SEOUT model and U 2 -Net_Se 2 Model, based on GF23B dataset for U 2 -Net_seIN model, U 2 -Net_SEOUT model and U2 -Net_Se 2 The model determines the optimal network model and selects the U with the highest accuracy and reliability. 2 -Net_Se 2 The model predicts the breeding areas and analyzes the prediction results.

[0049] Figure 2 The network structure diagram of the nested UNet neural network provided in the embodiment of the present invention, the overall structure of the nested UNet neural network presents a U-shaped structure, including an encoder, a decoder and a saliency map fusion module, the encoder is composed of six RSU modules from En_1 to En_6, the decoder is composed of five RSU modules from De_1 to De_5, the saliency map fusion module includes a saliency map fusion input module and a saliency map fusion output module, wherein the saliency map fusion input module is connected to the encoder, and the saliency map fusion output module is connected to the decoder. The saliency map fusion module adopts a gradually connected method to perform feature fusion, first collects the outputs of De_1, De_2, De_3, De_4, De_5 and En_6, and then passes through a convolution layer with a convolution kernel number of 1 and a convolution kernel size of 3x 3, respectively, to obtain a feature map with a layer number of 1. By bilinear interpolation, the feature map is scaled back to the input image size to obtain Sup1, Sup2, Sup3, Sup4, Sup5, and Sup6. Then, these 6 feature maps are concatenated to obtain a feature map with channel=6. Finally, a 1x 1 convolution layer and a Sigmiod activation function are used to obtain the final prediction probability map. Sup(n) represents the nth feature map that has been upsampled, Sup(0) represents the feature map output from the RSU module but has not been upsampled, and Sup(n) represents the feature map that has been upsampled. (n) Side indicates the nth division in the side direction. Figure 2 It can be seen that each RSU module in the nested UNet neural network also has a U-shaped structure, so the nested UNet neural network is composed of two layers of nested U-shaped structures. The RSU module has a symmetrical structure, and the number of layers on one side is the module height L. The green layer includes the convolution layer, normalization layer, and ReLU activation function, the blue layer is the downsampling layer, including the convolution layer, normalization layer, and ReLU activation function, the purple layer is the upsampling layer including the convolution layer, normalization layer, and activation function, and the gray layer is the green layer determined by the expansion coefficient.

[0050] By adding the channel attention module SE module based on global maximum pooling and average pooling inside and outside the structure of the En_1 to En_4 modules of the encoder part of the U-shaped structure of the nested UNet neural network, the U 2 -Net_Se 2 The model has a network structure like Figure 3 As shown in Figure 1, the SE module is located at the bottom of the En_1 to En_4 modules. Taking the En_1 module as an example, when the height L is 7, the internal structure of the RSU module after adding the channel attention module is as follows: Figure 4 As mentioned above, the SE module is located at the bottom of the blue layer, that is, the image data enters the SE module after downsampling. The added channel attention module based on global maximum pooling and average pooling is shown in Figure 5 As shown, x represents the original image data, h represents the result of the "compression" operation of the feature map, w represents the learned weight, and γ is a scaling parameter of the batch normalization layer, which is used to adjust the normalized feature value. Given an input image X with a feature channel number of C1, a feature map with a feature channel number of C is obtained through convolution processing, and then its dimension is compressed. The upper and lower branches are respectively subjected to global average pooling and global maximum pooling to obtain pooling results of size 1×1×C. The fully connected layer is used for processing, and the ReLU function is used as the activation function to obtain the upper and lower branch fully connected layer results of size 1×1×C / γ. The role of γ is to control the attention intensity and parameter amount. The upper and lower branches are processed by the fully connected layer and fused channel by channel. The Sigmoid activation function is used to compress the channel-by-channel fusion results to between [0.1] to obtain the final weight values ​​of each channel. Finally, each channel of the input feature map is multiplied by the channel weight value to obtain a feature map with channel weight information. Adding a second global maximum pooling branch on the basis of the original SE module can obtain richer global information, try to avoid the loss of too much information, make the channel weight information more convincing, and use these two features to greatly improve the representation ability of the network.

[0051] Containing global semantic information and greater The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A breeding area prediction method based on an improved nested UNet neural network, characterized in that: include: S1. Acquire Gaofen-3 image data of the breeding area through the Gaofen-3 satellite, and perform data preprocessing on the Gaofen-3 image data; S2. Obtain radar image data of the breeding area by resampling and image registration processing the pre-processed Gaofen-3 image data; S3. Obtaining Gaofen-2 image data of the breeding area through the Gaofen-2 satellite, performing data preprocessing on the Gaofen-2 image data, and obtaining optical image data of the breeding area; S4. Construct the GF23B dataset by performing band synthesis and image fusion on the radar image data and optical image data of the breeding area; S5. Based on the original UNet neural network model, the model complexity is improved by modifying the model network structure to obtain a nested UNet neural network; S6. Construct the U2-Net_Se2 neural network model by adding a channel attention mechanism to the nested UNet neural network; S7. Based on the GF23B dataset, the U2-Net_Se2 neural network model was trained and verified, and the prediction of breeding areas was realized through the U2-Net_Se2 neural network model.

2. The breeding area prediction method based on the improved nested UNet neural network according to claim 1, characterized in that: In S1, data preprocessing of GF-3 image data includes radiometric calibration, data conversion, multi-view processing, filtering processing and geocoding.

3. The breeding area prediction method based on the improved nested UNet neural network according to claim 2 is characterized in that: In S3, the data preprocessing of GF-2 image data includes radiometric calibration and atmospheric correction.

4. The breeding area prediction method based on the improved nested UNet neural network according to claim 3 is characterized in that: In S4, the radar image data of the breeding area is used as a band and the band is synthesized with the optical image data of the breeding area.

5. The breeding area prediction method based on the improved nested UNet neural network according to claim 4 is characterized in that: In S4, the image fusion algorithm NNDiffuse Pan Sharpening is used to perform pixel-level image fusion on the radar image data and the optical image data of the breeding area. The spatial and spectral features of the two image data are fused through image fusion, and the texture features of the image data are increased. The fused image data and the label data corresponding to the image data are uniformly cropped to a size of 512×512 to construct the GF23B dataset.

6. The breeding area prediction method based on the improved nested UNet neural network according to claim 5 is characterized in that: In S5, the nested UNet neural network is a two-layer nested U-shaped structure, including an encoder, a decoder and a saliency map fusion module. The encoder consists of six roadside unit RSU modules from En_1 to En_6, and the decoder consists of five roadside unit RSU modules from De_1 to De_5. The saliency map fusion module is connected to the encoder and the decoder, and feature fusion is performed in a gradually connected manner. The En_6 module of the encoder and the De_5 to De_1 modules of the decoder are subjected to 3×3 convolution and upsampling processing, and the output image is processed to be consistent with the size of the input image. After 1×1 convolution processing, the activation function Sigmoid function is used to generate a saliency probability map.

7. The breeding area prediction method based on the improved nested UNet neural network according to claim 6 is characterized in that: The RSU-L module is a highly determined RSU module, where L is the height, which is determined by the spatial resolution of the input feature. The height of the RSU module of the En_5 module is 7, and the height of the RSU module of the En_6 module is 4. Through pooling and upsampling operations, the RSU-7 module of the En_5 module and the RSU-4 module of the En_6 module are replaced with the RSU-4F module, where F represents extension. The resolution of the intermediate feature map is processed to be the same as the resolution of the input feature map through the RSU-4F module.

8. The breeding area prediction method based on the improved nested UNet neural network according to claim 7, characterized in that: In S6, the U2-Net_Se2 model is constructed by adding channel attention modules both inside and outside the U-shaped structure of the nested UNet neural network.

9. The breeding area prediction method based on the improved nested UNet neural network according to claim 8, characterized in that: The channel attention module is a SE module based on global maximum pooling and average pooling, which is located inside and outside the structure of the En_1 to En_4 modules of the encoder part of the U-shaped structure of the nested UNet neural network.

10. The breeding area prediction method based on the improved nested UNet neural network according to claim 9, characterized in that: In S7, the U2-Net_Se2 neural network model was trained based on the training set of the GF23B dataset. The spatial and spectral characteristics of the image data in the GF23B dataset were self-learned by the model to predict the area and distribution of the breeding area. The model was verified based on the validation set of the GF23B dataset to test the model prediction accuracy.

Images