An agricultural remote sensing image semantic segmentation method based on channel feature enhancement
By introducing channel feature enhancement and adaptive resampling training strategies into semantic segmentation of agricultural remote sensing images, the problems of insufficient utilization of multi-channel features and data imbalance in existing technologies are solved, thereby improving the prediction accuracy and robustness of the model.
Patent Information
- Application Number
- CN202211371489.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-03
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2042-11-03
AI Technical Summary
Existing IBN-Net and IBN-s modules cannot fully utilize multi-channel features in semantic segmentation of agricultural remote sensing images, and the training strategies cannot effectively alleviate the imbalance of samples in the dataset, resulting in insufficient prediction accuracy and robustness.
We employ a channel feature enhancement method, combined with an adaptive resampling training strategy and a weighted loss function. Through an improved IBN-a module and ResNet-CE model, we enhance feature extraction capabilities and achieve accurate predictions using a spatial pyramid structure and a decoder module.
The model improved the semantic class prediction accuracy and robustness of agricultural remote sensing images, significantly enhanced performance on imbalanced datasets, and achieved higher recognition accuracy.
Smart Images
Figure CN115909332B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of aerospace remote sensing technology, specifically a semantic segmentation method for agricultural remote sensing images based on channel feature enhancement. Background Technology
[0002] Deep learning has driven progress in multiple research areas across various visual tasks. Semantic segmentation, in particular, is a technique for separating different semantic classes from an image. A semantic segmentation model should be able to label the location and outline of a specific target. Visual tasks involving farmland have significant economic value, but progress in semantic segmentation methods applied to agricultural images has been minimal.
[0003] IBN-Net[1] is a module based on instance normalization (IN) and batch normalization (BN). IN can learn some features that will not change with appearance changes such as color and style, while BN can help preserve information related to the original features of the image and use it for subsequent feature extraction steps of the model. IBN-Net has good versatility and can be used as a module in different models, but it has shortcomings when applied to the field of agricultural remote sensing images. This technology is based on traditional images composed of three channels of red, green and blue, i.e., RGB images. However, agricultural remote sensing images may have multimodal features, such as near-infrared features, thus forming a multi-channel original feature map with more than three channels. This means that IBN-Net, which is based on RGB images, cannot make full use of these extended features. Then, the IBN-s[2] module was proposed, and IBN-Net was successfully applied to the semantic segmentation dataset of agricultural remote sensing images. It proposed a method to adjust IBN-Net by replacing IN and BN with SN. However, SN is still composed of IN and BN. It can only weigh the influence of IN and BN according to different model depths. This can reduce the KL divergence between RGB and near-infrared features, but the effect is limited. In fact, on different datasets, the effect is not as good as IBN-Net.
[0004] [1] Pan
[0005] [2]Yang S, Yu S, Zhao B, et al.Reducing the feature divergence of RGBand near-infrared images using Switchable Normalization[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern RecognitionWorkshops.2020:46-47. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a semantic segmentation method for agricultural remote sensing images based on channel feature enhancement. This method enables the model to predict the semantic class of each pixel more accurately, improving prediction precision and robustness.
[0007] The technical solution to achieve the objective of this invention is:
[0008] A semantic segmentation method for agricultural remote sensing images based on channel feature enhancement includes the following steps:
[0009] S1: Divide the agricultural semantic segmentation image dataset into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model's generalization performance.
[0010] S2: Define the preprocessing scheme for data reading, the specific process is as follows:
[0011] S21: Read the RGB image and the corresponding extended feature image respectively, read the corresponding label image from each semantic class and stitch them together. If a dataset contains a semantic classes, then the label image is an a-channel image.
[0012] S22: Randomly adjust the color saturation of the RGB image;
[0013] S23: Simultaneously perform random scaling, random vertical flipping, random horizontal flipping, and random rotation on RGB images, extended feature images, and label images;
[0014] S24: Standardize the feature matrices of the RGB image and the extended image, and concatenate the RGB image and the extended feature image by channel. If the number of channels of the extended feature image is b, then the extended feature image and the RGB image are concatenated to form an image with 3+b channels.
[0015] S3: Define a random weight sampler. A mature model requires multiple rounds of training, that is, each repetition of all steps is a round of training. In the first round of training of the model, assign the same weight to each image in the training set for random sampling. Define a distributed data loader to achieve the purpose of using multiple graphics cards for distributed training and improve the model training efficiency.
[0016] S4: Load data from the data loader and begin feature extraction. The specific steps are as follows:
[0017] S41: Pass the image feature matrix x and the label image matrix into the GPU;
[0018] S42: The image feature matrix x is mapped to a high-dimensional channel using a 2D convolutional layer with a kernel of 7, and then reaches the bottleneck block. In the bottleneck block, the W×H×C dimensional image feature matrix x is passed to the channel enhancement module, where W×H represents the number of pixels in each feature layer and C represents the number of channels. The features of each layer are compressed into a 1×1×C feature vector h by average pooling. cne ;
[0019] S43: For the eigenvector h cne Convolution is performed on a one-dimensional convolutional layer with a kernel size of 3 and an equal number of input and output channels to obtain the feature vector h1;
[0020] S44: Map the feature vector h1 through a fully connected layer whose input dimension is the original number of input channels of the image feature matrix x and whose output dimension is the number of output channels of the bottleneck block feature extraction. Then, use the new vector to cover and update the feature vector h1 to obtain the feature vector h2.
[0021] S45: Pass the feature vector h2 into the Sigmoid layer, use the new vector to overwrite and update the feature vector h3, and return it;
[0022] S46: Perform feature extraction on the original image feature matrix x before IBN-a residual connection, and update it to obtain the image feature matrix x1;
[0023] S47: Multiply the image feature matrix x1 and the feature vector h3 obtained in S45, update the image feature matrix x1 and perform residual connection, and return the overlay updated image feature matrix x2 to the main framework of the model after calculation through the ReLU layer;
[0024] S5: Using a spatial pyramid structure and a decoder module, the model's predicted image x for each pixel of the image is calculated. pred ;
[0025] S6: Calculate the loss function, update the iterative parameters using the gradient descent algorithm, and after multiple iterations, make the loss function converge. The loss function formula is defined as follows:
[0026] L total =0.5L CE +0.5L Dice ,
[0027] Where L CE L represents the cross-entropy loss function. Dice This represents the Dice loss function, which is obtained by weighting the two loss functions with the same weights to obtain the overall loss function.
[0028] S7: After model training is complete, the testing phase begins. The trained model is used to obtain the predicted image feature matrix x using the test set. val The steps are the same as the training process in steps S4 to S5, and the recognition accuracy (IoU) of the model for each category is calculated. i , where i represents a semantic class;
[0029] S8: According to the formula:
[0030]
[0031] Calculate the sampling probability for each semantic class, where n total n represents the number of all images in the current dataset. i Represents the number of images involved in the i-th semantic class, IoU i This represents the recognition accuracy of the current model for semantic class i. The IoU score is a standard performance metric for object class segmentation problems. Since an image contains multiple semantic classes, the probability of an image being sampled for training is defined as the average probability of all semantic classes involved in that image. The sampling data loader is redefined to train the model according to the sampling probability of each image.
[0032] S9: Steps S2 to S8 constitute one round of model training. Repeat steps S2 to S8 according to the predetermined goal to complete the model training.
[0033] Compared with existing technologies, this technical solution:
[0034] To address the issue that IBN-Net cannot fully utilize image features with four or more channels, a new technique is proposed that surpasses IBN-Net's performance under the same environmental configuration and experimental data. Furthermore, significant imbalance exists in agricultural remote sensing image semantic segmentation datasets, which IBN-Net's training strategy does not address. While IBN-s uses a weighted loss function to address sample imbalance, it is still insufficient. Therefore, based on a different weighted loss function than IBN-s, an adaptive resampling training strategy is proposed to mitigate the impact of imbalance and improve model performance.
[0035] This method effectively extracts channel feature information, enabling the model to pay differentiated attention to features located in different channels, focusing more on important information. Furthermore, the adaptive resampling training strategy involved in this method, combined with a weighted loss function, can effectively mitigate the negative impact of imbalanced samples in the dataset on model training, making the model's semantic class prediction for each pixel more accurate, thus improving prediction precision and robustness. Attached Figure Description
[0036] Figure 1 This is a structural diagram of the channel enhancement module introduced based on the IBN-a module in the embodiment;
[0037] Figure 2 This is a flowchart illustrating an example. Detailed Implementation
[0038] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.
[0039] Example:
[0040] This example replaces the original bottleneck block in the ResNet-101 model with, as shown below. Figure 1 The CE module shown is denoted as ResNet-CE, and ResNet-CE is used as an encoder in the DeepLabv3+ model. This example uses the Agriculture-Vision agricultural remote sensing semantic segmentation image dataset, which is a dataset taken by drones and contains 94,986 high-quality aerial images of 3,432 farmlands. Each set of images consists of RGB and near-infrared (NIR) channels.
[0041] like Figure 2 As shown, a semantic segmentation method for agricultural remote sensing images based on channel feature enhancement includes the following steps:
[0042] S1: Divide the agricultural semantic segmentation image dataset into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model's generalization performance.
[0043] S2: Define the preprocessing scheme for data reading, the specific process is as follows:
[0044] S21: Read the RGB image and the corresponding extended feature image respectively, read the corresponding label image from each semantic class and stitch them together. If a dataset contains 9 semantic classes, the label image is a 9-channel image.
[0045] S22: Randomly adjust the color saturation of the RGB image;
[0046] S23: Simultaneously perform random scaling, random vertical flipping, random horizontal flipping, and random rotation on RGB images, extended feature images, and label images;
[0047] S24: Standardize the feature matrices of the RGB image and the extended image, and concatenate the RGB image and the extended feature image by channel. If the extended feature image has 1 channel, the extended feature image and the RGB image will be concatenated to form a 4-channel image.
[0048] S25: Standardize the label images; In this example, the images in the Agriculture-Vision dataset are preprocessed. The RGB images with a dimension of 512×512×3 and the NIR images with a dimension of 512×512×1 are augmented and then stitched together to form a 512×512×4 feature map. The label images of multiple semantic classes are stitched together to form a new label map of 512×512×9.
[0049] S3: Define a random weight sampler. A complete model requires multiple rounds of training. That is, each repetition of all steps is a round of training. Determine whether the current round is the first round of training. If it is, define each image in the training set with the same sampling probability to construct a random weight sampler and a distributed data loader. This allows multiple graphics cards to be used for distributed training, improving the model training efficiency.
[0050] S4: Replace the original bottleneck block in the ResNet-101 model with, for example... Figure 1 The CE module shown is denoted as ResNet-CE. ResNet-CE is used as an encoder in the DeepLabv3+ model as the final model. The SGD optimizer is defined, with the learning rate set to 0.01 and the weight decay set to 0.0004. The defined model and optimizer are then passed to the GPU.
[0051] S5: Load data from the data loader and begin feature extraction. The specific steps are as follows:
[0052] S51: Construct a ResNet model. First, map the 512×512×4 scale image feature matrix to 512×512×64, and then complete the feature mapping through BatchNorm2d and ReLU layers to obtain the image feature matrix x.
[0053] S52: Image feature matrix x enters the bottleneck block, such as Figure 1 As shown, the image feature matrix x is fed into the channel enhancement module to obtain the channel feature vector h. cne After the image feature matrix x undergoes IBN-a feature processing, a new feature matrix x1 is obtained. x1 is then compared with the channel feature vector h. cneAfter multiplication, the updated feature matrix x2 is obtained. The residual connection adds x2 to the original features and passes it through a ReLU layer to obtain the final image feature matrix x. final And return;
[0054] S53: Since ResNet-CE and ResNet-101 have the same overall structure, feature extraction is performed by repeating S52 according to the model structure of ResNet-101.
[0055] S54: Convert the image feature matrix x obtained in steps S51 to S53 into... final Following the framework design of the DeepLabv3+ model, the ASPP module and decoder module are input, ultimately yielding the model prediction graph x. pred The scale is 512×512×9, consistent with the scale of the label image, and returns the matrix x. pred ;
[0056] S6: Calculate the loss function, update the iterative parameters using the gradient descent algorithm, and after multiple iterations, make the loss function converge. The loss function formula is defined as follows:
[0057] L total =0.5L CE +0.5L Dice ,
[0058] Where L CE L represents the cross-entropy loss function. Dice This represents the Dice loss function, which is obtained by weighting the two loss functions with the same weights to obtain the overall loss function.
[0059] S7: After model training is complete, the testing phase is initiated to verify the model's generalization performance. The trained model is used to obtain the predicted image feature matrix x using the test set. val The steps are the same as the training process from S4 to S5, and the IoU of the model's recognition accuracy for each category is calculated. i , where i represents a semantic class;
[0060] S8: According to the formula:
[0061]
[0062] Calculate the sampling probability for each semantic class, where n total n represents the total number of images in the current dataset. i Represents the number of images involved in the i-th semantic class, IoU iThis indicates the accuracy of the current model in recognizing semantic class i. The IoU score is a standard performance metric for object class segmentation problems. Since an image contains multiple semantic classes, the probability of an image being sampled for training is defined as the average probability of all semantic classes involved in that image. Based on the sampling probability of each image, the sampling data loader is redefined to perform a new round of training on the model.
[0063] S9: The entire training process involves 2000 warm-start training sessions, 8000 static learning rate training sessions, and 20000 learning rate decay training sessions. At this point, the loss function tends to converge, and training stops.
[0064] By implementing other existing methods under the same environmental configuration, the recognition accuracy was obtained for comparison. The experimental results are shown in Table 1:
[0065] method mIoU (%) LinkNet 29.62 DeepLabv3+ 38.41 FPN 38.61 DeepLabv3 41.14 IBN-s 41.39 IBN-a 45.29 This method 47.01
[0066] As can be seen from Table 1, the recognition accuracy of the method in this example is higher than that of existing methods.
Claims
1. An agricultural remote sensing image semantic segmentation method based on channel feature enhancement, comprising the following steps: S1: dividing an agricultural semantic segmentation image dataset, dividing the data into a training set and a test set; S2: defining a preprocessing scheme in data reading, the specific process being: S21: reading the RGB image and the corresponding extended feature image respectively, reading the corresponding label image from each semantic class and splicing them together, if a certain dataset contains a semantic class, then the label image is an a-channel image spliced from the a semantic class label image; S22: randomly adjusting the color saturation of the RGB image; S23: simultaneously performing random scaling, random vertical flipping, random horizontal flipping, and random rotation on the RGB image, the extended feature image, and the label image; S24: normalizing the RGB image and the extended feature image matrix, splicing the RGB image and the extended feature image by channel, if the number of channels of the extended feature image is b, then the extended feature image and the RGB image are spliced to form a 3+b-channel image; S3: defining a random weight sampler, a shaped model needs multiple rounds of training, that is, each repetition of all steps is a round of training, in the first round of training of the model, the same weight is assigned to each image in the training set for random sampling, a distributed data loader is defined, so that multiple graphics cards can be used for distributed training, and the model training efficiency is improved; S4: loading data from the data loader and starting feature extraction, the specific steps being as follows: S41: passing the image feature matrix x and the label image matrix into the GPU; S42: After the image feature matrix x is mapped to a high-dimensional channel using a two-dimensional convolutional layer with a convolution kernel of 7, the image feature matrix x of WxHxC dimensions (where WxH represents the number of pixel points contained in each layer of features, and C represents the number of channels) is transmitted to a channel enhancement module in a bottleneck block, and each layer of features is subjected to average pooling compression to form a feature vector h of 1x1xC. cne ; S43: obtaining a feature vector h cne Convolution is performed on a one-dimensional convolution layer with a kernel size of 3, an equal number of input channels and output channels, to obtain a feature vector h1. S44: mapping the feature vector h1 in the input dimension of the original input channel number of the image feature matrix x to the output dimension of the bottleneck block feature extraction output channel number of the full connection layer, and using the new vector to cover and update the feature vector h1 to obtain the feature vector h2; S45: passing the feature vector h2 into the Sigmoid layer, using the new vector to cover and update to obtain the feature vector h3 and returning; S46: performing feature extraction on the original image feature matrix x before IBN-a residual connection, and covering and updating to obtain the image feature matrix x1; S47: multiplying the image feature matrix x1 and the feature vector h3 obtained in S45, updating the image feature matrix x1 and residual connection, and returning to cover and update the image feature matrix x2 after ReLU layer calculation to the backward model main framework; S5: using the spatial pyramid structure and the decoder module, calculating a prediction map x of each pixel point of the image by the model pred ; S6: calculate the loss function, update the iterative parameters using the gradient descent algorithm, and make the loss function converge after multiple iterations, wherein, The loss function formula is defined as: L total = 0.5L CE + 0.5L Dice , wherein L CE represents a cross-entropy loss function, L Dice represents a Dice loss function, and the overall loss function is obtained by combining the two loss functions with the same weight. S7: After the model training ends, enter the testing link, and use the test set to obtain the predicted image feature matrix x of the trained model val The steps are consistent with the training process of steps S4 to S5, and the recognition accuracy IoU of the model for each category is calculated i i represents a semantic class; S8: according to the formula: Compute the sampling probability of each semantic class, where n total represents the number of all images in the current data set, n i represents the number of images involved in the i-th semantic class, IoU i represents the recognition accuracy of the current model for semantic class i, the IoU score is a standard performance measure for object class segmentation problems, since a picture contains multiple semantic classes, the probability that a picture is sampled for training is defined as the average probability of all semantic classes involved in the picture, according to the sampling probability of each picture, the sampling data loader is redefined to train the model; S9: steps S2 to S8 are a round of model training, and steps S2 to S8 are repeated according to the predetermined target to complete the model training.