Rice remote sensing image segmentation method based on multi-scale converter

Abstract

The invention relates to the technical field of semantic segmentation, in particular to a rice remote sensing image segmentation method based on a multi-scale converter, and solves the problems that in the prior art, remote sensing affects a semantic segmentation process and depends on a self-attention mechanism, and the multi-scale feature integration capacity is poor. The method comprises the following steps: S1, making a data set; s2, designing and building a training model; s3, training, and iterating to network convergence to obtain a semantic segmentation model; and S4, taking the newly added remote sensing images of different months as the input of the trained semantic segmentation model, and performing reasoning through the model to obtain the remote sensing image after plot segmentation. The semantic segmentation model based on the self-adaptive multi-scale hybrid converter is designed, the precision is improved through the feature fusion module and the multi-layer optimization decoder module, the robustness and adaptability are enhanced, and the method has high practical value for agricultural monitoring and related application.

Description

Technical Field

[0001] The invention relates to the technical field of semantic segmentation, and in particular to a rice remote sensing image segmentation method based on a multi-scale transformer. Background Art

[0002] Traditional computer vision semantic segmentation methods are usually based on low-level features of images such as color, texture, and edges, combined with some classic machine learning algorithms and techniques.

[0003] Traditional semantic segmentation algorithms mainly rely on artificially designed features and algorithms. They are difficult to handle some complex scenes and high-dimensional features and are easily affected by lighting and noise, resulting in poor segmentation effects. Their adaptability and generalization capabilities are also weak.

[0004] Deep learning semantic segmentation is based on deep neural network technology through end-to-end learning to classify images at the pixel level. These deep learning models can classify objects more accurately and draw precise boundaries to define different categories through analysis at the granular level. They can also automatically learn high-level features of images, reduce manual feature design, and better adapt to complex scenes and diverse image data through end-to-end training models. At present, deep learning has made remarkable achievements in semantic segmentation, but the existing algorithms will lose some spatial information after hollow convolution or simple pooling, especially in capturing the global information of large-scale targets and the detailed information of small targets at the same time. In addition, some deep learning models will lose many details and edge information of features through multiple downsampling operations on high-level feature maps, which makes it impossible to locate and segment the edges of targets more accurately.

[0005] In 2022, Wang et al. proposed the paper "Masked-attention Mask Transformer for Universal Image Segmentation", which proposed three major improvements based on Maskformer. One is to change the attention added in the transformer decoder to mask attention and exchange positions with self attention to ensure that the foreground acquires features from each other and ignores the interference of the background; the other is to add multi-scale features, that is, the different layers of the pixel decoder are input into the different layers of the transformer decoder as feature pyramids to improve the recognition effect of small targets; finally, the mask loss is calculated on a small number of sampled random points instead of the entire image to improve the speed. The focus of the optimization part is that in the first part, in the Mask2Former network, the role of the mask attention module is to learn the dependency between the current position and other positions through the attention mechanism, while considering the mask information in the input sequence. This module can help the model better understand and process the input when there are missing values ​​in the processing sequence, while reducing the computational complexity of the model. Therefore, the Mask Attention module can help improve the robustness and generalization ability of the model and improve the model performance. In the second part, the High-resolution features module is proposed, which is used to extract high-resolution features from low-resolution feature maps. Since low-resolution feature maps have excellent performance in semantic information, but low resolution and poor performance in target area details, they need to be converted into high-resolution feature maps through the High-resolution features module to better capture the details of the target area. This module uses convolution and deconvolution operations to increase the resolution of feature maps by increasing the width and height of feature maps. In this way, the Mask2Former network can detect and segment targets more accurately and improve the accuracy of the model. Finally, an optimization strategy is proposed. The optimizer improvement module (Optimization improvements) mainly optimizes the network training process and improves the training speed and accuracy of the model. The optimizer improvement module (Optimization improvements) in the Mask2Former network includes two improvements.On the one hand, a new learning rate adjustment strategy is used, namely the learning rate adjustment strategy based on gradient momentum, which can better control the size of the learning rate and the change of the gradient, thereby improving the stability and convergence speed of training; on the other hand, a layered adaptive momentum based batch training (LAMB) optimizer based on the Adam optimization algorithm is introduced, which can better solve the problem of gradient disappearance or explosion and improve the training effect of the network.

[0006] Mask2Former relies on the self-attention mechanism to capture the global information of the input image, but in remote sensing images, the boundaries between the background and objects, such as buildings, roads, and vegetation, are often fuzzy, especially in low-resolution or noisy remote sensing data. Although the Transformer architecture has the ability to model global information, it is difficult to effectively handle the subtle differences between the background and the target, especially in the case of long distances or small targets. It performs poorly when facing complex backgrounds and is prone to mis-segmentation or missed segmentation, which reduces the accuracy and reliability of the model.

[0007] Remote sensing images usually contain multi-scale information, and different areas in the image may have different spatial resolutions and features. For example, the detailed information of urban areas and the features of large-scale natural environments are very different. When processing remote sensing images of different scales, Mask2Former may find it difficult to capture local details and global information at the same time, especially in multi-scale and complex scenes, where its performance may decline. Although the Transformer model has advantages in modeling long-distance dependencies, it has a weak ability to integrate multi-scale features, which is particularly prominent in tasks such as remote sensing images with multi-level and multi-scale features. In addition, remote sensing data usually comes from different sensors and shooting angles, resulting in large differences in image spectral, texture and other features. The model has poor robustness and generalization ability when processing these diverse data. Although Mask2Former can perform detailed image segmentation, it may be limited by different data sets and tasks in the face of these changes, and it is difficult to efficiently adapt to new remote sensing image data, resulting in limited scalability of the model, especially when facing cross-regional data.

[0008] Annotated data in remote sensing images are usually difficult to obtain. The annotation of remote sensing data not only requires expert knowledge, but also may cause problems with the quality and consistency of the annotated data due to differences in image features from different sensors, different shooting angles, and different seasons. Therefore, when processing training data, it relies heavily on annotations, which may lead to poor performance with a small amount of annotated data. In the actual application of remote sensing images, image data generated by different regions and different sensors are often encountered, and the distribution of these data is often different from the training data, which limits the generalization ability of the model. Especially in tasks such as change detection and land use classification in remote sensing images, the model cannot be well migrated to new environments, affecting its performance in diverse application scenarios.

[0009] There is an urgent need for a new remote sensing image semantic segmentation method that can solve the above problems. Summary of the invention

[0010] The rice remote sensing image segmentation method based on multi-scale transformer proposed in the present invention solves the problems of relying on self-attention mechanism and poor multi-scale feature integration ability in the remote sensing semantic segmentation process in the prior art.

[0011] The technical solution of the present invention is implemented as follows: a rice remote sensing image segmentation method based on a multi-scale transformer is characterized in that it comprises the following steps:

[0012] S1. Create a dataset;

[0013] S2. Design and build a training model: Input the manually annotated images in S1 into the semantic segmentation model for training;

[0014] The training model is a semantic segmentation model, including a backbone network for predicting possible overlapping binary mask predictions and function activations; a feature hybrid network module that represents the relationship between the same category by calculating the similarity between the intermediate feature blocks and the original image blocks to form a feature matrix; and a multi-layer optimized decoder module that fuses the feature images of three different scales of the decoder for output;

[0015] S3. Perform training and iterate until the network converges to obtain a semantic segmentation model; the deep learning framework used is PyTorch; the Focal Loss and Dice Loss mixed loss function is used in training 0;

[0016] S4. Add the remote sensing images of different months as the input of the trained semantic segmentation model, perform inference through the model, and obtain the remote sensing images after segmenting the plots.

[0017] In step S2, the backbone network specifically extracts features from the image to generate a low-resolution image feature map. Among them, CF represents the number of channels, S is the stride of the feature map; the pixel decoder is used to gradually upsample the image features to extract pixel by pixel; the standard Transformer decoder is used to focus on image features and learnable position embeddings and use a feature pyramid composed of low resolution and high resolution, and the resolution of the feature pyramid is returned to the Transfomrer decoder layer at one time to generate N per-segment embeddings, which independently generate class probability prediction values ​​of N classes The model predicts N possible overlapping binary mask predictions through the dot product between pixel-level embeddings and mask embeddings; then uses the sigmoid function for activation, that is, m i [h,w]=sigmoid(ε mask [:,i] T ·ε pixel [:,h,w]).

[0018] The feature hybrid network module in step S2 takes the encoder original feature map F1 and the third layer feature map F3 as input to form a similarity matrix, i.e., a feature fusion module. Given a color image I∈R W×H×3 As input, the encoder of the segmentation model uses the swin transformer to extract feature maps {F i |i=1,...,L}, where and Wi, Hi, Ci represent the width, height and number of channels of the feature map respectively; specifically:

[0019] The first step is to use a 7*7 convolutional layer With 64 filters and a 3*3 convolutional layer With 1 filter for embedding the first level feature map, i.e., color image I, R img ∈R H×W×1 , for the i-th layer feature map Use a 1*1 convolutional layer with N c The filters transform the feature map of the i-th level into a feature map for each category N c is the number of categories to be predicted; assuming Contains only relevant information for the kth category;

[0020] The second step is to convert R img Split into patches of fixed size, this process is expressed as P img =φ(R img ,d),In this process,φ is the image partition operation, R img ∈R H×W×1Makes a series of flat 2D patches Where P img The jth vector of Contains a patch of resolution d*d of the original color image, and also ith Each channel of is divided into a series of flat patches, and the process formula is expressed as Here yes A sequence of 2D patches, where || denotes concatenation.

[0021] The third step is to img and P ith Each of The dot product method is used to determine the relationship between each patch of the original image and the k-th category-related feature map. The formula for this process is expressed as Here Represents the association between each image patch of the original image and each image patch of the k-th class-related feature map;

[0022] The attention matrix F will be obtained ith The output part of the decoder uses a multi-layer optimized decoder module to generate the encoder result. Use a fixed-size average pooling layer to compress the decoder output and R ith The same size, this process formula is expressed as

[0023] Here ψ(input,k H ,k W ,s H ,s W ) is the kernel and stride size of the pooling layer, which is the same as the original image and feature map. mask Each channel of is divided into a series of flat patches. This process is expressed as Here The obtained attention matrix F ith Applied to P mask To adjust the output probability of the encoder, the formula is expressed as

[0024]

[0025] Here and Because P mask The mask contains the predicted mask of the compressed image patch, F ith It represents the relationship between the initial image patch and the feature image patch. It shows the effect of the large object output on the small object;

[0026] Finally Fold to the final result M mask The same size, the prediction result is Here

[0027] The input feature map is convolved sideways through two convolutional layers to generate an intermediate feature map with the same resolution as the input feature map. Subsequently, after further convolution, the generated feature map is used in the subsequent top-down approach to add the lower resolution feature map to the higher resolution feature map through upsampling.

[0028] A fusion mechanism of global context information is introduced: the last layer of feature maps is processed by a global average pooling operation and adaptively upsampled to the same size as the output feature map; then, the processed global feature map is multiplied with the feature map generated by the feature pyramid network. The above process can be expressed by the following formula: Output i is the output feature map of the i-th layer, lateral i is the side feature map of the current layer.

[0029] The loss function in step S3 is specifically implemented as:

[0030] By introducing two hyperparameters, modulation factor α and focusing factor γ, on the basis of the standard cross entropy loss, the standard cross entropy loss is defined as: CE(p t )=-log(p t ), where p t is the model's predicted probability for the sample, and Focal Loss is defined as: FL(p t )=-α t (1-p t ) γ log(p t ), where α t represents the modulation factor used to balance positive and negative samples, γ represents the focusing factor used to control the weight of easy-to-classify samples, and p t is the model's predicted probability for the sample;

[0031] By optimizing the Dice coefficient, the overlap between the model's prediction results and the true label is maximized, and by modifying the weights of positive and negative samples, it can be applied to the unbalanced segmentation task. The coefficient is defined as:

[0032] Where A and B are the two sets of prediction results and true labels in the segmentation task, TP represents the number of pixels correctly predicted as positive, FP represents the number of pixels incorrectly predicted as positive, and FN represents the number of pixels incorrectly predicted as negative.

[0033] By combining the two loss functions, we get our mask loss definition:

[0034] L mask (m,m gt )=λ focal L focal (m,m gt )+λ dice L dice (m,m gt ), and the hyperparameter λ focal =20.0 and λ dice =1.0.

[0035] The invention discloses a rice remote sensing image segmentation method based on a multi-scale transformer, designs a semantic segmentation model based on an adaptive multi-scale hybrid transformer, performs feature fusion by using original image features and feature maps in a backbone network process through a feature fusion module, represents the similarity of objects of the same category by calculating the relationship matrix between intermediate feature blocks and image blocks, and utilizes the relationship to enhance the information loss of multi-scale targets of the same category; fuses feature images of three different scales of the decoder to form a final output through a multi-layer optimized decoder module, enhances the expression ability of network features by integrating local and global feature information, and expands the receptive field to enhance the expression ability of edge details; not only improves the accuracy, but also enhances the robustness and adaptability, and has high practical value for agricultural monitoring and related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 is a flow chart of the method of the present invention;

[0038] Figure 2 A technical roadmap for the method of the present invention;

[0039] Figure 3 A network structure diagram of the method of the present invention;

[0040] Figure 4 A multi-layer optimized decoder module of the present invention;

[0041] Figure 5 Schematic diagram of the segmentation effect of rice remote sensing image. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The rice remote sensing image segmentation method based on a multi-scale transformer disclosed in the present invention is characterized in that it comprises the following steps:

[0044] S1. Create a dataset;

[0045] S2. Design and build a training model: Input the manually annotated images in S1 into the semantic segmentation model for training;

[0046] The training model is a semantic segmentation model, including a backbone network for predicting possible overlapping binary mask predictions and function activations; a feature hybrid network module that represents the relationship between the same category by calculating the similarity between the intermediate feature blocks and the original image blocks to form a feature matrix; and a multi-layer optimized decoder module that fuses the feature images of three different scales of the decoder for output;

[0047] S3. Perform training and iterate until the network converges to obtain a semantic segmentation model. The deep learning framework used is PyTorch. A mixed loss function of focal loss and dice loss is used in training. Focal loss is used to deal with the problem of class imbalance and dice loss is used to optimize the overlap of segmentation results, thereby improving the accuracy of the model. The optimizer uses Adam for training, and the learning rate is set to 1×10 -5 , the number of iterations is 500000;

[0048] S4. Add the remote sensing images of different months as the input of the trained semantic segmentation model, perform inference through the model, and obtain the remote sensing images after segmenting the plots.

[0049] In step S2, the backbone network specifically extracts features from the image to generate a low-resolution image feature map. Where H represents the length of the image, W represents the width of the image, and C Frepresents the number of channels, S is the stride of the feature map; the image features are gradually upsampled using a pixel decoder to extract pixel by pixel; the image feature map and the learnable position are embedded using a standard Transformer decoder and a feature pyramid composed of low-resolution and high-resolution feature maps is used to return the resolution of the feature pyramid to the transformer decoder layer at one time, generating N per-segment embeddings, which independently generate class probability prediction values ​​for N classes The model predicts N possible overlapping binary mask predictions through the dot product between the pixel-level embedding and the mask embedding; then uses the sigmoid function for activation, that is, m i [h,w]=sigmoid(ε mask [:,i] T ·ε pixel [:,h,w]).

[0050] The feature hybrid network module in step S2 takes the encoder original feature map F1 and the third layer feature map F3 as input to form a similarity matrix, i.e., a feature fusion module. Given a color image I∈R W×H×3 As input, the encoder of the segmentation model uses the Swin transformer to extract feature maps {F i |i=1,...,L}, where Wi, Hi, Ci represent the width, height and number of channels of the feature map respectively; specifically:

[0051] The first step is to use a 7*7 convolutional layer With 64 filters and a 3*3 convolutional layer With 1 filter for embedding the first level feature map, i.e., color image I, R img ∈R H×W×1 , for the i-th layer feature map Use a 1*1 convolutional layer with N c The filters transform the feature map of the i-th level into a feature map for each category N c is the number of categories to be predicted; assuming Contains only relevant information for the kth category;

[0052] The second step is to convert R img Split into patches of fixed size, this process is expressed as P img =φ(R img ,d),In this process,φ is the image partition operation, R img ∈R H×W×1 Makes a series of flat 2D patches Where P img The jth vector of Contains a patch of resolution d*d of the original color image, and also ith Each channel of is divided into a series of flat patches, and the process formula is expressed as Here yes A sequence of 2D patches, where || denotes concatenation.

[0053] The third step is to img and P ith Each of The dot product method is used to determine the relationship between each patch of the original image and the k-th category-related feature map. The formula for this process is expressed as Here Represents the association between each image patch of the original image and each image patch of the k-th class-related feature map;

[0054] The attention matrix F will be obtained ith The output part of the decoder is used to optimize and enhance the segmentation of these small objects; the encoder results using multiple layers of optimized decoder modules Use a fixed-size average pooling layer to compress the decoder output and R ith The same size, this process formula is expressed as

[0055] Here ψ(input,k H ,k W ,s H ,s W ) is the kernel and stride size of the pooling layer, which is the same as the original image and feature map. mask Each channel of is divided into a series of flat patches. This process is expressed as Here The obtained attention matrix F ith Applied to P mask To adjust the output probability of the encoder, the formula is expressed as

[0056]

[0057] Here and Because P mask The mask contains the predicted mask of the compressed image patch, F ith It represents the relationship between the initial image patch and the feature image patch. It shows the effect of the large object output on the small object;

[0058] Finally Fold to the final result M mask The same size, the prediction result is Here These steps can effectively enhance the model's sensitivity to small targets. By considering similar imaging features of the same object category without increasing the size of the dataset or expanding the image features, the feature relationship between objects of the same size can be used to compensate for the loss of feature information caused by feature propagation.

[0059] The multi-layer optimized decoder module in step S2 is specifically:

[0060] A feature pyramid network is implemented, which performs side convolution operations on the input feature map through two convolutional layers to generate an intermediate feature map with the same resolution as the input feature map; then, after further convolution processing, the generated feature map is used for subsequent feature extraction and fusion;

[0061] In the process of constructing multi-scale feature maps, a top-down approach is adopted to add lower-resolution feature maps to higher-resolution feature maps through upsampling operations, thereby achieving multi-scale feature fusion. This layer-by-layer fusion strategy can effectively integrate information of different scales and improve the performance of the model in multi-scale object detection.

[0062] A fusion mechanism of global context information is introduced: the last layer of feature maps is processed by global average pooling operation, and it is adaptively upsampled to the same size as the output feature map; then, the processed global feature map is multiplied with the feature map generated by the feature pyramid network to achieve the fusion of global context information. This fusion strategy of global and local information enhances the model's perception of different scales and context information, thereby improving the resolution and expression of feature maps. The above process can be expressed by the following formula: Output i is the output feature map of the i-th layer, lateral i is the side feature map of the current layer. In the process of generating multi-scale feature maps, the feature map of each layer is multiplied with the global pooling feature map and added to the side convolution result of the current layer before being added to the multi-scale feature map list. The list saves feature maps from low to high resolution for subsequent segmentation tasks, ensuring that the input image can be effectively represented at multiple scales, enhancing the segmentation capability of objects of different scales.

[0063] The loss function in step S3 is specifically implemented as:

[0064] The focal loss function mainly adjusts the weights of easy-to-classify samples and difficult-to-classify samples so that the model can pay more attention to the difficult-to-classify sample data during training. By introducing two hyperparameter modulation factors α and focusing factor γ on the basis of the standard cross entropy loss, the standard cross entropy loss is defined as: CE(p t )=-log(p t ), where p t is the model's predicted probability for the sample, and Focal Loss is defined as: FL(p t )=-α t (1-p t ) γ log(p t ), where α t represents the modulation factor used to balance positive and negative samples, γ represents the focusing factor used to control the weight of easy-to-classify samples, and p t is the model's predicted probability for the sample;

[0065] The Dice Loss function is mainly used to measure the similarity between two sets of data. It maximizes the overlap between the model's prediction results and the true labels by optimizing the Dice coefficient. It can also be applied to unbalanced segmentation tasks by modifying the weights of positive and negative samples. The coefficient is defined as:

[0066] Where A and B are the two sets of prediction results and true labels in the segmentation task, TP represents the number of pixels correctly predicted as positive, FP represents the number of pixels incorrectly predicted as positive, and FN represents the number of pixels incorrectly predicted as negative.

[0067] By combining the two loss functions, we get our mask loss definition:

[0068] L mask (m,m gt )=λ focal L focal (m,m gt )+λ dice L dice (m,m gt ), and the hyperparameter λ focal =20.0 and λ dice =1.0.

[0069] The reference indicators used in the present invention are PA and IoU: Pixel accuracy (PA) is a common indicator for measuring the performance of image segmentation models, which is used to evaluate the proportion of each pixel correctly classified by the model in the image segmentation task. The calculation method of PA is very simple: it is the ratio of the number of all correctly classified pixels to the total number of pixels. The higher the value of PA, the better the model is at predicting the correct category of each pixel. In practical applications, PA is an intuitive indicator, especially in classification tasks, which can show the accuracy of the model's prediction of each pixel. Intersection over Union (IoU) is another important indicator for evaluating the performance of image segmentation models, which measures the degree of overlap between the predicted segmentation area and the true annotated area. IoU reflects the ratio of the intersection and union between the area predicted by the model and the true annotated area, and is one of the criteria for measuring segmentation quality. The higher the value of IoU, the greater the overlap between the model's prediction result and the true annotation, and the better the segmentation effect. The range of IoU values ​​is from 0 to 1, where 0 means no overlap at all and 1 means complete overlap. Unlike pixel accuracy (PA), IoU is more sensitive to class imbalance problems. When there are fewer target classes and more background pixels, PA may give a higher score, while IoU can more accurately reflect the performance of the model in processing target class segmentation.

[0070] Step S4: Adding rice remote sensing images of different months as input of the trained semantic segmentation model, performing inference through the model, and obtaining the segmented rice plots. Figure 5 The middle left image is used as the input of the model, and the model is inferred after the model is inferred. Figure 5 In the middle right figure, it can be seen that the trained model can segment the rice part in the remote sensing image. We use the evaluation indicators PA and IoU to prove the effectiveness of this method. This method is compared with Maskformer and Mask2Former, and the results are all trained under the same experimental conditions, that is, a single Tesla A100 GPU. Compared with the original model method, the PA and IoU indicators of this invention have been improved, proving the effectiveness of this method.

[0071] Table 1: Comparison of objective evaluation indicators of text dataset experimental results

[0072] method PA IOU Params(M) Maskformer 58.9 77.7707 63M Mask2Former 61.7 78.875 65M Ours 66.9 80.0893 51M

[0073] The embodiment uses a remote sensing image dataset specific to the Chengdu area, and all data are manually annotated. This dataset has strong regional specificity and can truly reflect the rice planting environment and its complexity in the Chengdu area, such as seasonal changes, climatic conditions, and the diversity of rice growth. By annotating and training local data, the model can better adapt to the unique geographical, climatic and crop planting characteristics of the Chengdu area, thereby significantly improving the accuracy and precision of the rice segmentation task. This localized dataset also helps to reduce the errors that may be caused by external datasets and improve the practical application value of the segmentation model in the region, especially in the actual deployment of agricultural fields such as agricultural monitoring and precision fertilization, which can better meet user needs and enhance the generalization ability of the model.

[0074] The embodiment mainly aims at the rice segmentation problem in remote sensing images, and designs an optimized deep learning semantic segmentation method, aiming to improve the accurate segmentation ability of rice in remote sensing images, especially the performance when processing multi-scale targets. As an important target in agricultural monitoring, rice often appears in remote sensing images of a large range, and its segmentation accuracy directly affects the application effects in the fields of crop monitoring, yield estimation and pest warning. In order to deal with the multi-scale problem that may occur in rice segmentation, the present invention first designs a method for preparing a remote sensing image data set specifically for rice. The data set ensures the diversity and representativeness of the data by annotating rice images at different growth stages, different geographical regions, and different lighting and meteorological conditions. The annotation process adopts a fine pixel-level annotation method, and focuses on annotating the performance of rice at different scales, providing rich training samples for the deep learning model. In terms of deep learning model optimization, the present invention adopts a multi-scale feature fusion network, which can accurately capture the detailed features of rice at different growth stages and different distribution ranges through effective multi-level feature extraction and fusion strategies. By introducing a feature fusion module, the method effectively fuses the original image features and the intermediate layer features, and enhances the model's ability to capture multi-scale information. This not only improves the segmentation accuracy of rice areas, but also effectively reduces the interference of background noise, especially when dealing with small and dense areas. The multi-layer optimized decoder module optimizes the information flow of the decoder by integrating local and global features, and improves the robustness of the model in rice boundary details and complex environments. In addition, this method also performs well in capturing targets at different scales, effectively avoiding the problem of target loss, enhancing the adaptability of the model, and being able to cope with different scenes and changes in complex remote sensing images.

[0075] The invention discloses a rice remote sensing image segmentation method based on a multi-scale transformer, designs a semantic segmentation model based on an adaptive multi-scale hybrid transformer, performs feature fusion by using original image features and feature maps in a backbone network process through a feature fusion module, represents the similarity of objects of the same category by calculating the relationship matrix between intermediate feature blocks and image blocks, and utilizes the relationship to enhance the information loss of multi-scale targets of the same category; fuses feature images of three different scales of the decoder to form a final output through a multi-layer optimized decoder module, enhances the expression ability of network features by integrating local and global feature information, and expands the receptive field to enhance the expression ability of edge details; not only improves the accuracy, but also enhances the robustness and adaptability, and has high practical value for agricultural monitoring and related applications.

[0076] Of course, without departing from the spirit and essence of the invention, technicians familiar with the field should be able to make various corresponding changes and modifications based on the invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the invention.

Claims

1. A rice remote sensing image segmentation method based on a multi-scale transformer, characterized in that: Including semantic segmentation model, which is a segmentation model based on adaptive multi-scale hybrid transformer; The following steps are involved: S1. Create a dataset; S2. Design and build a training model: Input the manually annotated images in S1 into the semantic segmentation model for training; The training model is a semantic segmentation model, including a backbone network for predicting possible overlapping binary mask predictions and function activations; a feature hybrid network module that represents the relationship between the same category by calculating the similarity between the intermediate feature blocks and the original image blocks to form a feature matrix; and a multi-layer optimized decoder module that fuses the feature images of three different scales of the decoder for output; S3. Perform training and iterate until the network converges to obtain a semantic segmentation model; the deep learning framework used is PyTorch; the Focal Loss and Dice Loss mixed loss function is used in training; S4. Add the remote sensing images of different months as the input of the trained semantic segmentation model, perform inference through the model, and obtain the remote sensing images after segmenting the plots.

2. The rice remote sensing image segmentation method based on multi-scale transformer according to claim 1, characterized in that: In step S2, the backbone network specifically extracts features from the image to generate a low-resolution image feature map. Among them, C F represents the number of channels, S is the stride of the feature map; the pixel decoder is used to gradually upsample the image features to extract pixel by pixel; the standard Transformer decoder is used to focus on image features and learnable position embeddings and use a feature pyramid composed of low resolution and high resolution, and the resolution of the feature pyramid is returned to the transformer decoder layer at one time to generate N per-segment embeddings, which independently generate class probability prediction values ​​for N classes The model predicts N possible overlapping binary mask predictions through the dot product between the pixel-level embedding and the mask embedding; then uses the sigmoid function for activation, that is, m i [h,w]=sigmoid(ε mask [:,i] T ·ε pixel [:,h,w]).

3. The rice remote sensing image segmentation method based on multi-scale transformer according to claim 1, characterized in that: The feature hybrid network module in step S2 takes the encoder original feature map F1 and the third layer feature map F3 as input to form a similarity matrix, i.e., a feature fusion module; Given a color image I∈R W×H×3 As input, the encoder of the segmentation model uses the swin transformer to extract feature maps {F i |i=1,...,L}, where and Wi, Hi, Ci represent the width, height and number of channels of the feature map respectively; specifically: The first step is to use a 7*7 convolutional layer Conv 1 1Conv1 with 64 filters and a 3*3 convolutional layer 2 With 1 filter for embedding the first level feature map, i.e., color image I, R img ∈R H×W×1 , for the i-th layer feature map Use a 1*1 convolutional layer with N c The filters transform the feature map of the i-th level into a feature map for each category N c is the number of categories to be predicted; assuming (1≤k≤N c ) contains only relevant information about the kth category; The second step is to convert R img Split into patches of fixed size, this process is expressed as P img =φ(R img ,d),In this process,φ is the image partition operation, R img ∈R H×W×1 Makes a series of flat 2D patches Where P img The jth vector of Contains a patch of resolution d*d of the color original image, and also ith Each channel of is divided into a series of flat patches, and the process formula is expressed as Here (1≤k≤N c )yes A sequence of 2D patches, where || denotes concatenation, The third step is to img and P ith Each of The dot product method is used to determine the relationship between each patch of the original image and the k-th category-related feature map. The formula for this process is expressed as Here Represents the association between each image patch of the original image and each image patch of the k-th class-related feature map; The attention matrix F will be obtained ith For the output part of the decoder; Encoder results using multiple layers of optimized decoder modules (1≤k≤N c ), using a fixed-size average pooling layer to compress the decoder output and R ith The same size, this process formula is expressed as Here ψ(input,k H ,k W ,s H ,s W ) is the kernel and stride size of the pooling layer, which is the same as the original image and feature map. mask Each channel of is divided into a series of flat patches. This process is expressed as Here The obtained attention matrix F ith Applied to P mask To adjust the output probability of the encoder, the formula is expressed as Here and Because P mask The mask contains the predicted mask of the compressed image patch, F ith It represents the relationship between the initial image patch and the feature image patch. It shows the effect of the large object output on the small object; Finally Fold to the final result M mask The same size, the prediction result is Here Pre∈R H×W×Nc .

4. The rice remote sensing image segmentation method based on multi-scale transformer according to claim 1, characterized in that: The multi-layer optimized decoder module in step S2 is specifically: The input feature map is subjected to side convolution operations through two convolutional layers to generate an intermediate feature map with the same resolution as the input feature map; then, after further convolution processing, the generated feature map is used for subsequent feature extraction and fusion; In a top-down approach, lower-resolution feature maps are added to higher-resolution feature maps through upsampling operations; A fusion mechanism of global context information is introduced: the last layer of feature maps is processed by a global average pooling operation and adaptively upsampled to the same size as the output feature map; then, the processed global feature map is multiplied with the feature map generated by the feature pyramid network. The above process can be expressed by the following formula: Output i is the output feature map of the i-th layer, lateral i is the side feature map of the current layer.

5. The rice remote sensing image segmentation method based on multi-scale transformer according to claim 1, characterized in that: The loss function in step S3 is specifically implemented as: By introducing two hyperparameters, modulation factor α and focusing factor γ, on the basis of the standard cross entropy loss, the standard cross entropy loss is defined as: CE(p t )=-log(p t ), where p t is the model's predicted probability for the sample, and Focal Loss is defined as: FL(p t )=-α t (1-p t ) γ log(p t ), where α t represents the modulation factor used to balance positive and negative samples, γ represents the focusing factor used to control the weight of easy-to-classify samples, and p t is the model's predicted probability for the sample; By optimizing the Dice coefficient, the overlap between the model's prediction results and the true label is maximized, and by modifying the weights of positive and negative samples, it can be applied to the unbalanced segmentation task. The coefficient is defined as: Where A and B are the two sets of prediction results and true labels in the segmentation task, TP represents the number of pixels correctly predicted as positive, FP represents the number of pixels incorrectly predicted as positive, and FN represents the number of pixels incorrectly predicted as negative. By combining the two loss functions, we get our mask loss definition: L mask (m,m gt )=λ focal L focal (m,m gt )+λ dice L dice (m,m gt ), and the hyperparameter λ focal =20.0 and λ dice =1.0.

Images