A rice field weed semantic segmentation method based on multispectral unmanned aerial vehicle image

Abstract

The present application relates to a kind of rice field weed semantic segmentation method based on multispectral unmanned aerial vehicle image.It solves the problem of poor precision of general semantic segmentation technology in prior art for rice field weed identification and segmentation.It includes constructing physical-data dual-driven joint attention mechanism, and designing multi-modal joint loss function as unified back propagation constraint in training pipeline, with very high interpretability and theoretical rigor, solving the problem of accurate segmentation of small target and boundary of rice field weed under extremely similar morphology.The present application has the advantages of: solving the technical problems of insufficient utilization of multispectral remote sensing information by conventional network and low band semantic differentiation, improving the model's ability to perceive subtle spectral differences of crops;Improve feature robustness under complex field interference, completely improve the defects of traditional model such as small target missed detection, boundary segmentation burr and low classification accuracy.

Description

Technical Field

[0001] This invention relates to the fields of agriculture and computer vision technology, specifically to a semantic segmentation method for weeds in rice paddies based on multispectral UAV images. Background Technology

[0002] As a core grain crop in my country, large-scale and precise field management is crucial for ensuring stable and increased rice production. Weeds in paddy fields compete with cultivated rice for sunlight, water, fertilizer, and growing space, easily causing lodging, yield reduction, and quality decline. This is a significant field disaster hindering the large-scale, high-quality, and high-yield cultivation of rice. Field weeds are similar in plant type, texture, and appearance to cultivated rice, making manual identification difficult and inspection inefficient. Traditional management methods, such as manual weeding and large-scale pesticide spraying, are not only labor-intensive and inefficient, but also prone to secondary problems such as pesticide residues, damage to the paddy field ecosystem, and pesticide damage to rice.

[0003] With the rapid development of UAV remote sensing and deep learning technologies, crop semantic segmentation technology based on aerial imagery has been widely applied in smart agriculture scenarios such as field target recognition, vegetation classification, and pest and disease monitoring, providing technical support for the automated identification and precise positioning of weeds in paddy fields. Current mainstream segmentation models mostly employ general convolutional networks or Transformer architectures, relying on data-driven approaches to complete feature learning and pixel classification. However, they still have significant technical shortcomings in complex paddy field scenarios: First, paddy field scenarios suffer from extremely unbalanced sample distribution, with cultivated rice pixels accounting for a very high proportion, while weeds are distributed as sparse small targets and scattered patches. General models tend to learn the features of the main cultivated rice, resulting in missed or false detections of weeds. Second, the visual features of weeds and cultivated rice are highly similar, and relying solely on visible light texture features has extremely low distinguishability. Existing models fail to fully explore the vegetation physiological and spectral differences contained in multispectral bands, resulting in low utilization of spectral features. Third, the pixel mixing in crop boundary areas is severe, and edge features are blurred. Conventional loss functions lack boundary-specific constraints, leading to jagged, sticky, and poor boundary accuracy in segmentation contours. Fourth, existing attention mechanisms mostly focus on local spatial features or single-channel features, lacking the ability to model the global distribution features of the paddy field scene. They cannot effectively identify aberrant weed targets that deviate from the distribution of the main cultivated rice, and the robustness of small-target weed identification is poor.

[0004] In summary, existing general semantic segmentation techniques are ill-suited to the challenges of identifying sparsely distributed, mixed-boundary, and subtly spectral weeds in paddy fields. They suffer from shortcomings such as insufficient utilization of spectral information, lack of global feature modeling, weak small target recognition capabilities, low boundary segmentation accuracy, and poor generalization, failing to meet the demands of large-scale precision weeding and intelligent inspection in paddy fields. Therefore, a multispectral weed semantic segmentation method that integrates remote sensing physical priors and global scene feature modeling is urgently needed to address the core deficiencies of existing technologies. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned technical problems by providing a semantic segmentation method for rice paddy weeds based on multispectral UAV images. Breaking away from the purely data-driven limitations of traditional black-box networks, a physical-data dual-driven joint attention mechanism (PGMMA) is constructed. Furthermore, a multimodal joint loss function is designed in the training pipeline as a unified backpropagation constraint. This method, with its high interpretability and theoretical rigor, solves the challenge of accurately segmenting small targets and boundaries of rice paddy weeds with extremely similar morphologies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a semantic segmentation method for weeds in rice paddies based on multispectral UAV images, comprising the following steps:

[0007] S1. Use drones to collect raw multispectral images of rice paddies and perform preprocessing.

[0008] S2, Multispectral Input and Multiscale Feature Extraction;

[0009] S3, Physical Information Guided Joint Attention Module (PGMMA) for fine-grained execution;

[0010] S4. Constraints on the multimodal joint loss function during backpropagation;

[0011] S5. Comparative and ablation experiments.

[0012] In step S1, the preprocessing step includes:

[0013] S1-1, sequentially complete radiometric calibration to correct image radiometric errors, vignetting correction to correct lens edge brightness attenuation, noise removal to filter out salt and pepper, streaks, and noise, and geometric registration to correct image distortion and offset.

[0014] S1-2 integrates the three bands of RGB visible light with four types of characteristic spectral bands: multispectral narrow band red light, multispectral narrow band green light, near-infrared band, and red edge band, and reconstructs a complete image input tensor containing seven bands;

[0015] S1-3. The fixed input size of the image input tensor is: H×W×C, 512×512×7, where 512 are the image height and width dimensions respectively, and 7 is the number of image bands.

[0016] In step S2, the image input tensor preprocessed in step S1 is input into the SSTB backbone network for independent feature extraction and global modeling. Specific steps include:

[0017] S2-1, Band Independent Convolutional Encoder:

[0018] For each band b (b ∈ {1, 2, ..., B}, B=7), local feature extraction is performed using a depthwise convolutional group:

[0019] , ∈ R^(H×W×C);

[0020] in: For the input of band b, DWConv is an encoding module containing 3 depthwise separable convolutional layers and batch normalization and GELU activation functions, with C=64 output channels;

[0021] S2-2, Multi-band feature stitching and fusion:

[0022] All independent features of all bands are spliced ​​along the channel dimension to form a fused feature tensor:

[0023] F_concat = Concat[F1, F2, ..., F7] ∈ R^(H×W×(C×7));

[0024] Cross-band channel compression is performed using 1×1 convolution, with an output dimension of R^(H×W×D) and D=256.

[0025] S2-3, Spectral-Spatial Transformer encoder:

[0026] The fused feature maps are flattened into a sequence and input into the encoder for progressive processing of the fused convolutional features, generating multi-scale feature maps. The encoder consists of 12 Spectral-Spatial Transformer layers, each containing:

[0027] A. Multi-head self-attention: 6 attention heads are used to capture long-range dependencies between bands;

[0028] B. Feedforward network: Two-layer MLP with a scaling ratio of 4, used for feature transformation;

[0029] C. Learnable spectral position coding: Independently learns position coding for each band to distinguish band semantics;

[0030] D. Residual connectivity and layer normalization: ensure the stability of feature propagation.

[0031] In steps S2-3, the encoder distributes the 12 Transformer layers evenly across four stages, with each stage containing 3 stacked Transformer layers:

[0032] Stage 1 (Shallow Features): The input features undergo preliminary processing, including a 4x downsampling, to output a shallow feature map with a size of 128×128×256. ;

[0033] Stage 2 (Mid-Shallow Features): Processes the downsampled features from Stage 1, performing an 8x downsampling to output a mid-shallow feature map with a size of 64×64×256. ;

[0034] Stage 3 (Medium-Deep Features): Processes the downsampled features from Stage 2, performing a 16x downsampling to output a medium-deep feature map with a size of 32×32×256. ;

[0035] Stage 4 (Deep Features): Processes the downsampled features from Stage 3, performing a 32x downsampling to output a deep feature map with a size of 16×16×256. .

[0036] In step S3, the Physical Information Guided Joint Attention Module (PGMMA) completes adaptive weighted optimization driven by the collaboration of physical rules and network mechanisms during forward propagation. It adopts a dual-input architecture, with one input receiving the high-level spatial semantic feature map output from the SSTB backbone network. ~ One path serves as the data-driven feature stream; the other directly inputs the original multispectral image tensor. Real remote sensing spectral information is used as the source of physical prior constraints.

[0037] Step S3 specifically includes the following steps:

[0038] S3-1, Global Scene Deviation Spatial Attention (GSDA) Submodule

[0039] The Global Scene Deviation Spatial Attention (GSDA) submodule mines heterogeneous target regions based on global feature statistical differences. It quantifies the degree of feature deviation between pixels and the overall scene through cosine deviation, accurately highlighting the outlier features of weeds. Global average pooling is used to aggregate the spatial information of the entire image to obtain a semantic center vector representing the overall distribution of the rice paddy scene.

[0040] ;

[0041] In the formula, H=128 and W=128 are the feature map space dimensions, and the mean feature of this semantic center vector is biased towards the main cultivated rice feature distribution.

[0042] Using the global mean as a benchmark, the feature vector similarity deviation is calculated pixel by pixel, and the deviation value represents the degree of pixel heterogeneity:

[0043] ;

[0044] A deviation value close to 0 indicates that the pixel features fit the mainstream distribution of cultivated rice; a deviation value close to 2 indicates that the pixel is a sparse outlier target that deviates from the overall scene, corresponding to the weedy area in the field.

[0045] The bias features are learned and the weights are scaled using 1×1 pointwise convolution, and then normalized to the effective weight range using Sigmoid activation. The output generates a global bias space attention map.

[0046] ;

[0047] S3-2, Vegetation Index Enhanced Attention (VIGA) Submodule

[0048] The Vegetation Index Enhanced Attention (VIGA) submodule explicitly extracts near-infrared band features corresponding to the backbone network. Features of the red-edge band A lightweight two-stage downsampling module is constructed, consisting of two layers of 3×3 depthwise separable convolutions with a stride of 2, BN and GELU activation functions. Under the premise of low parameter quantity, high-resolution physical features are uniformly mapped to a 128×128 scale, and the aligned physical feature stream is output.

[0049] Based on the remote sensing physical characteristics of the vegetation red edge effect, and relying on the aligned original physical features, an end-to-end differentiable implicit vegetation index operator is constructed:

[0050] ;

[0051] ;

[0052] In the formula, , The network is designed to learn adaptive scalar parameters, with an initial value of 1.0. These parameters can dynamically correct residual errors in radiometric calibration of UAV images and spectral nonlinear deviations caused by environmental interference during backpropagation. The Sigmoid activation function is used to normalize the attention weight range. This is a point-by-point convolution used to integrate local pixel differences and optimize weight distribution;

[0053] S3-3, High-Efficiency Channel-Spatial Concatenated Attention (EC-SA) Submodule

[0054] The efficient channel-space concatenated attention (EC-SA) submodule optimizes feature representation from both global channel and local pixel space dimensions, capturing refined geometric features of sparse weeds in rice paddies with extremely low parameter count; the refined geometric features are then fed into the EC-SA submodule for unidirectional concatenated attention enhancement.

[0055] S3-4, the three-multiplication fusion mechanism of the right-side convergence node

[0056] The right-side convergence node integrates spectral optimization features, channel spatial enhancement features, and physical prior constraint features using a multi-weighted cascaded product fusion method, achieving multi-dimensional feature adaptive weighted fusion. The weighted fusion formula is as follows:

[0057] ;

[0058] In the formula, This is a high-level semantic feature map output by the backbone network;

[0059] Channel attention weight map;

[0060] SA is a spatial attention weight map;

[0061] A vegetation physical attention map driven by the original image;

[0062] For global bias attention.

[0063] In step S3-3, the execution steps of unidirectional serial attention enhancement include:

[0064] S3-31. First, through the channel attention mechanism, relying on the cascaded global max pooling and average pooling combined with a lightweight MLP layer, a 1×1×256-dimensional channel attention weight map CA is generated to complete the suppression of redundant features in the channel dimension and the enhancement of effective features.

[0065] S3-32. The optimized features are concatenated and fed into the spatial attention mechanism. A 7×7 large receptive field convolutional layer captures the geometric features of a large range of connected regions, generating a 128×128×1 dimension spatial attention weight map SA, which realizes the local spatial region feature focusing and localization.

[0066] Step S4 specifically includes the following steps:

[0067] S4-1. Constructing a multimodal joint loss function Guided gradient iteration:

[0068] ;

[0069] S4-2, Weighted Multimodal Dice Loss ( )

[0070] ;

[0071] In the formula: This represents the foreground prediction probability corresponding to pixel i.

[0072] The corresponding binary label;

[0073] Set pixel weight rules: the foreground pixel weight for weeds is 3.0, and the background pixel weight is 1.0; It is a smoothing constant;

[0074] S4-3, Boundary Distance Transformation Loss ( )

[0075] ;

[0076] In the formula: N is the total number of pixels in the image;

[0077] To predict the pixel features of the mask boundary;

[0078] It is a distance feature map obtained by transforming the real label mask with Euclidean distance and normalizing it;

[0079] S4-3, Small Target Enhancement Loss ( )

[0080] For connected components in an image whose pixel projection area is less than a preset threshold τ (τ=256 pixels), we define them as a sparse set of small targets S, and apply a focus loss constraint to them individually:

[0081] ;

[0082] Among them, the preferred value of the focus factor is... ,

[0083] |S| represents the total number of pixels within the small target set.

[0084] In step S5, the comparative experiment steps include:

[0085] S5-1. Perform precise pixel-by-pixel annotation by hand, and divide the training set, validation set, and test set according to a ratio of 7:1.5:1.5;

[0086] S5-2. Conduct quantitative comparative experiments with mainstream baseline methods on the exact same hardware environment and test set;

[0087] The ablation experiment involves gradually disassembling or assembling core sub-modules while maintaining complete consistency of the backbone network.

[0088] The hardware environment uses PyTorch 2.0 and CUDA 11.8; the network optimizer is AdamW, the base learning rate is set to 1×10^-3, the weight decay coefficient is 1×10^-4, the learning rate scheduler uses cosine annealing, and the batch size is configured to 4.

[0089] Compared with the prior art, the advantages of the present invention are as follows:

[0090] 1. This invention constructs the SSTB spectral-spatial Transformer backbone network to achieve independent encoding of multi-band spectra and long-range dependency modeling. Unlike traditional single-scale and single-channel feature extraction methods, it fully explores the spectral differences and spatial multi-scale features of multispectral images, solves the technical problems of insufficient utilization of multispectral remote sensing information and low band semantic discrimination of conventional networks, and greatly improves the model's ability to perceive subtle spectral differences in crops.

[0091] 2. The PGMMA multi-dimensional joint attention module designed in this invention achieves bidirectional fusion of physical priors and data-driven approaches. The four attention mechanisms have clear division of labor and complete positive interaction and complementarity. The newly added GSDA global scene bias attention, for the first time, uses global feature cosine bias to mine sparse and heterogeneous weed targets in paddy fields, breaking through the limitation of traditional local convolutional attention being unable to perceive global distribution differences; the VIGA module introduces remote sensing vegetation physics laws, using the original multispectral bands as hard constraints to correct the problem of deep feature spectral distortion; the EC-SA module achieves lightweight enhancement in both channel and spatial dimensions, accurately capturing the geometric details of weeds, and the multi-mechanism collaboration significantly improves the feature robustness under complex field disturbances.

[0092] 3. This invention constructs a multimodal joint loss function system to specifically address the industry pain points of extremely imbalanced rice paddy samples, mixed boundaries, and sparse small targets. Weighted Dice loss balances the distribution of foreground and background samples, boundary distance transformation loss specifically optimizes the crop boundary ambiguity problem, and small target focus loss strengthens the learning of tiny weed features. This triple loss joint constraint thoroughly improves the shortcomings of traditional models, such as missed detection of small targets, jagged boundary segmentation, and low classification accuracy. The various indicators of this invention significantly outperform existing mainstream segmentation algorithms. It can be widely applied to intelligent inspection of rice paddies, providing efficient and reliable technical support for intelligent rice planting and refined field management, and possesses extremely high engineering application value and promotion prospects. Attached Figure Description

[0093] Figure 1 This is a flowchart of the method of the present invention;

[0094] Figure 2 This is a schematic diagram of the spectral-spatial Transformer backbone network in this invention;

[0095] Figure 3 This is a flowchart of the Physical Information Guided Joint Attention Module (PGMMA) and multi-track feature product in this invention. Detailed Implementation

[0096] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0097] like Figure 1-3 As shown, a semantic segmentation method for weeds in rice paddies based on multispectral UAV images includes the following steps:

[0098] S1. Use drones to collect raw multispectral images of rice paddies and perform preprocessing.

[0099] S2, Multispectral Input and Multiscale Feature Extraction;

[0100] S3, Physical Information Guided Joint Attention Module (PGMMA) for fine-grained execution;

[0101] S4. Constraints on the multimodal joint loss function during backpropagation;

[0102] S5. Comparative and ablation experiments.

[0103] The overall network architecture of the method of this invention consists of two main stages: using the Spectral-Spatial Transformer Backbone (SSTB) as the front-end feature extractor to fully model multi-band spectral information and spatial features;

[0104] The Physics-Guided Multi-Modal Attention (PGMMA) module serves as the core of the physics-data dual-driven attention enhancement, integrating spectral mutual information, vegetation index physical priors, and spatial-channel attention.

[0105] In step S1, the preprocessing step includes:

[0106] S1-1, sequentially complete radiometric calibration to correct image radiometric errors, vignetting correction to correct lens edge brightness attenuation, noise removal to filter out salt and pepper, streaks, and noise, and geometric registration to correct image distortion and offset.

[0107] S1-2 integrates the three bands of RGB visible light with four types of characteristic spectral bands: multispectral narrow band red light, multispectral narrow band green light, near-infrared band, and red edge band, and reconstructs a complete image input tensor containing seven bands;

[0108] S1-3. The fixed input size of the image input tensor is: H×W×C, 512×512×7, where 512 are the image height and width dimensions respectively, and 7 is the number of image bands.

[0109] In step S2, the image input tensor preprocessed in step S1 is input into the SSTB backbone network for independent feature extraction and global modeling. Specific steps include:

[0110] S2-1, Band Independent Convolutional Encoder:

[0111] For each band b (b ∈ {1, 2, ..., B}, B=7), local feature extraction is performed using a depthwise convolutional group:

[0112] , ∈ R^(H×W×C);

[0113] in: For the input of band b, DWConv is an encoding module containing 3 depth-separable convolutional layers (3×3 kernels, stride of 1) and batch normalization (BN) and GELU activation function, with an output channel number C=64.

[0114] S2-2, Multi-band feature stitching and fusion:

[0115] All independent features of all bands are spliced ​​along the channel dimension to form a fused feature tensor:

[0116] F_concat = Concat[F1, F2, ..., F7] ∈ R^(H×W×(C×7));

[0117] Cross-band channel compression is performed using 1×1 convolution, with an output dimension of R^(H×W×D) and D=256.

[0118] S2-3, Spectral-Spatial Transformer encoder:

[0119] The fused feature maps are flattened into a sequence and input into the encoder for progressive processing of the fused convolutional features, generating multi-scale feature maps. The encoder consists of 12 Spectral-Spatial Transformer layers, each containing:

[0120] A. Multi-Head Self-Attention (MHSA): 6 attention heads used to capture long-range dependencies between bands;

[0121] B. Feed-Forward Network (FFN): A two-layer MLP with a scaling factor of 4, used for feature transformation;

[0122] C. Learnable Spectral Positional Encoding: Independently learns positional encoding for each band to distinguish band semantics;

[0123] D. Residual connectivity and layer normalization: ensure the stability of feature propagation.

[0124] In steps S2-3, the encoder distributes the 12 Transformer layers evenly across four stages (Stage 1-Stage 4), with 3 Transformer layers stacked within each stage:

[0125] Stage 1 (Shallow Features): The input features undergo preliminary processing, including a 4x downsampling, to output a shallow feature map with a size of 128×128×256. ;

[0126] Stage 2 (Mid-Shallow Features): Processes the downsampled features from Stage 1, performing an 8x downsampling to output a mid-shallow feature map with a size of 64×64×256. ;

[0127] Stage 3 (Medium-Deep Features): Processes the downsampled features from Stage 2, performing a 16x downsampling to output a medium-deep feature map with a size of 32×32×256. ;

[0128] Stage 4 (Deep Features): Processes the downsampled features from Stage 3, performing a 32x downsampling to output a deep feature map with a size of 16×16×256. .

[0129] The output of each stage comes from the last Transformer layer of that stage. Step-by-step downsampling is used to extract multi-scale spatial features. After the multi-scale feature map set is output, it is designed to balance shallow texture and deep semantic information. , , Upsampled to a high resolution of 128×128 using bilinear interpolation, and... Alignment is performed to complete multi-scale feature fusion and dimensional unification, ultimately generating a high-resolution aggregated feature map F with stronger representation capabilities. Its size is 128×128×256, which serves as the standard input data for the subsequent PGMMA module.

[0130] In step S3, the Physical Information Guided Joint Attention Module (PGMMA) completes adaptive weighted optimization driven by the collaboration of physical rules and network mechanisms during forward propagation. It adopts a dual-input architecture, with one input receiving the high-level spatial semantic feature map output from the SSTB backbone network. ~ One path serves as the data-driven feature stream; the other directly inputs the original multispectral image tensor. Using real remote sensing spectral information as a physical prior constraint source, this approach avoids the spectral information distortion and feature-semantic misalignment issues caused by deep downsampling. Spectral features are scale-aligned to feature maps via a lightweight downsampling module. After processing, it participates in subsequent feature enhancement operations. Figure 3 Based on features The implementation method will be described. , , The number of downsampling layers differs in VIGA, the purpose of which is to keep the output and input feature dimensions consistent. The output results of 4 feature layers are accumulated after upsampling by bilinear interpolation.

[0131] Step S3 specifically includes the following steps:

[0132] S3-1, Global Scene Deviation Spatial Attention (GSDA) Submodule

[0133] Given the high proportion of cultivated rice and the sparse, anomalous distribution of weeds in paddy fields, this module utilizes the Global Scene Deviation Spatial Attention (GSDA) submodule to mine heterogeneous target regions based on global feature statistical differences. Cosine deviation is used to quantify the degree of feature deviation between pixels and the overall scene, accurately highlighting the anomalous features of weeds. Global average pooling is then employed to aggregate the spatial information of the entire image, yielding a semantic center vector representing the overall distribution of the paddy field scene.

[0134] ;

[0135] In the formula, H=128 and W=128 are the feature map space dimensions, and the mean feature of this semantic center vector is biased towards the main cultivated rice feature distribution.

[0136] Using the global mean as a benchmark, the feature vector similarity deviation is calculated pixel by pixel, and the deviation value represents the degree of pixel heterogeneity:

[0137] ;

[0138] A deviation value close to 0 indicates that the pixel features fit the mainstream distribution of cultivated rice; a deviation value close to 2 indicates that the pixel is a sparse outlier target that deviates from the overall scene, corresponding to the weedy area in the field.

[0139] The bias features are learned and the weights are scaled using 1×1 pointwise convolution, and then normalized to the effective weight range using Sigmoid activation. The output generates a global bias space attention map.

[0140] ;

[0141] This attention filter selects outlier targets at the global distribution level, making up for the deficiency that local convolutions cannot perceive differences in global features.

[0142] S3-2, Vegetation Index Enhanced Attention (VIGA) Submodule

[0143] To incorporate physical principles as guidance, the Vegetation Index Enhanced Attention (VIGA) submodule explicitly extracts the near-infrared band features corresponding to the backbone network. Features of the red-edge band To achieve pixel-level alignment between the original physical bands and the backbone semantic features, a lightweight two-stage downsampling module is constructed, consisting of two layers of 3×3 depthwise separable convolutions with a stride of 2, BN and GELU activation functions. Under the premise of low parameter quantity, high-resolution physical features are uniformly mapped to a 128×128 scale, and the aligned physical feature stream is output.

[0144] Based on the remote sensing physical characteristics of the vegetation red edge effect, and relying on the aligned original physical features, an end-to-end differentiable implicit vegetation index operator is constructed:

[0145] ;

[0146] ;

[0147] In the formula, , The network is designed to learn adaptive scalar parameters, with an initial value of 1.0. These parameters can dynamically correct residual errors in radiometric calibration of UAV images and spectral nonlinear deviations caused by environmental interference during backpropagation. The Sigmoid activation function is used to normalize the attention weight range. This is a point-by-point convolution used to integrate local pixel differences and optimize weight distribution;

[0148] The final generated dimension is Physical response attention map The weights are derived entirely from the physical spectral information of the original images, possessing a rigorous remote sensing physical foundation. To avoid abrupt weight changes and gradient oscillations in the early stages of training, a residual flow factor is constructed. This smoothly delivers the physical constraint features to subsequent fusion nodes.

[0149] S3-3, High-Efficiency Channel-Spatial Concatenated Attention (EC-SA) Submodule

[0150] The efficient channel-space concatenated attention (EC-SA) submodule optimizes feature representation from both global channel and local pixel space dimensions, capturing refined geometric features of sparse weeds in rice paddies with extremely low parameter count; the refined geometric features are then fed into the EC-SA submodule for unidirectional concatenated attention enhancement.

[0151] S3-4, the three-multiplication fusion mechanism of the right-side convergence node

[0152] The right-side convergence node integrates spectral optimization features, channel spatial enhancement features, and physical prior constraint features using a multi-weighted cascaded product fusion method, achieving multi-dimensional feature adaptive weighted fusion. The weighted fusion formula is as follows:

[0153] ;

[0154] In the formula, This is a high-level semantic feature map output by the backbone network;

[0155] Channel attention weight map;

[0156] SA is a spatial attention weight map;

[0157] A vegetation physical attention map driven by the original image;

[0158] For global bias attention.

[0159] The four types of attention enhance features from four independent dimensions: channel, local space, physical spectrum, and global distribution. They exhibit no functional overlap, strong complementarity, and ultimately enhance the physical information of the output features. The image is fed into the Unet standard semantic segmentation decoder and then upsampled using stepwise bilinear interpolation to restore it to the original resolution. , , , , The output results are accumulated after bilinear interpolation upsampling, and finally pixel-by-pixel classification is completed through the Softmax activation layer to output a high-precision weed prediction mask, achieving accurate segmentation and recognition in complex paddy field scenarios.

[0160] In step S3-3, the execution steps of unidirectional serial attention enhancement include:

[0161] S3-31. First, through the channel attention mechanism, relying on the cascaded global max pooling and average pooling combined with a lightweight MLP layer, a 1×1×256-dimensional channel attention weight map CA is generated to complete the suppression of redundant features in the channel dimension and the enhancement of effective features.

[0162] S3-32. The optimized features are concatenated and fed into the spatial attention mechanism. A 7×7 large receptive field convolutional layer captures the geometric features of a large range of connected regions, generating a 128×128×1 dimension spatial attention weight map SA, which realizes the local spatial region feature focusing and localization.

[0163] After outputting the prediction mask through forward inference, this invention constructs a multimodal joint loss function to alleviate problems such as uneven sample distribution, blurred crop boundary segmentation, and missed detection of small weeds. Guide gradient iteration.

[0164] Step S4 specifically includes the following steps:

[0165] S4-1. Constructing a multimodal joint loss function Guided gradient iteration:

[0166] ;

[0167] S4-2, Weighted Multimodal Dice Loss ( )

[0168] ;

[0169] In the formula: This represents the foreground prediction probability corresponding to pixel i.

[0170] The corresponding binary label;

[0171] To balance the difference in pixel count between foreground weeds and background, a pixel weighting rule is set: the foreground pixel weight for weeds is 3.0, and the background pixel weight is 1.0. This is a smoothing constant to avoid zero values ​​in the denominator during calculation.

[0172] S4-3, Boundary Distance Transformation Loss ( )

[0173] ;

[0174] In the formula: N is the total number of pixels in the image;

[0175] To predict the pixel features of the mask boundary;

[0176] It is a distance feature map obtained by transforming the real label mask with Euclidean distance and normalizing it;

[0177] The closer a pixel is to the boundary of the crop, the higher its feature value. The feature values ​​of the inner crop area and the outer background, which are far from the boundary, gradually decrease. This guides the network to prioritize optimizing the segmentation contour of the boundary area and improve the precision of edge segmentation.

[0178] S4-3, Small Target Enhancement Loss ( )

[0179] For connected components in an image whose pixel projection area is less than a preset threshold τ (τ=256 pixels), we define them as a sparse set of small targets S, and apply a focus loss constraint to them individually:

[0180] ;

[0181] Among them, the preferred value of the focus factor is... ,

[0182] |S| represents the total number of pixels within the small target set.

[0183] This scheme optimizes the loss weight configuration. , In practical applications, parameters can be flexibly fine-tuned based on the dataset sample distribution and image resolution.

[0184] In step S5, the dataset used was acquired by a multispectral UAV through multiple flights at a rice planting base. A total of 2734 paired high-resolution images with 7 complete spectral bands were obtained. The comparative experimental steps included:

[0185] S5-1. Perform precise pixel-by-pixel annotation by hand, and divide the training set (1914 images), validation set (410 images), and test set (410 images) according to a ratio of 7:1.5:1.5.

[0186] S5-2. Conduct quantitative comparative experiments with mainstream academic baseline methods on the exact same hardware environment and test set.

[0187] Evaluation metrics included mean intersection-union ratio (mIoU), F1-score, precision, recall, and recall for small targets with an area less than 256 pixels. The comparison results are shown in Table 1.

[0188] UNet 69.4 71.8 73.6 70.1 58.1 DeepLabV3+ 71.9 74.2 75.7 72.8 61.3 Segformer 76.5 78.4 79.8 77.1 63.2 Method of the present invention 81.3 83.5 85.2 81.9 74.5

[0189] Table 1. Quantitative comparison results of different methods on the multispectral rice intercropping weed test set.

[0190] As shown in Table 1, the method of this invention outperforms the comparison methods in all evaluation metrics. The mIoU is improved by 11.9% compared to the classic benchmark method UNet, and it still outperforms the commonly used Transformer structure Segformer by 4.8%. In terms of recall rate for sparse small targets that are extremely difficult to identify, this invention achieves 74.5%, surpassing all general methods.

[0191] To further decouple and demonstrate the independent contribution of each innovative sub-module, a module ablation study was conducted.

[0192] The ablation experiment involved the step-by-step disassembly or reassembly of core sub-modules while maintaining complete consistency of the backbone network. The experimental results are shown in Table 2.

[0193] SSTB trunk × × × × 72.3 58.4 Ablation Experiment 1 √ × × × 77.6 64.3 Ablation Experiment 2 √ √ × × 79.2 65.1 Ablation Experiment 3 √ √ √ × 79.8 68.4 Complete method of the present invention √ √ √ √ 81.3 74.5

[0194] Table 2 Ablation Experiment Results of the Core Innovative Sub-module of this Invention

[0195] The ablation experiment results clearly show that: after introducing GSDA, mIoU is improved by 5.3%, proving that global scene bias spatial attention can significantly improve classification purity; after incorporating the VIGA remote sensing physical prior learnable implicit operator, the performance increases monotonically by 1.6%, fully demonstrating the strong corrective and guiding effect of agricultural physical laws on deep networks; most importantly, after adding the inverse constraint of the multimodal joint loss function, the recall rate of small targets has increased by 6.1%, strongly demonstrating that distance transformation and focus constraint have a strong targeting suppression optimization ability for sparse small targets.

[0196] The model training and testing of the method of this invention were carried out on a high-performance server equipped with four NVIDIA A100 graphics cards. The hardware environment used PyTorch 2.0 and CUDA 11.8; the network optimizer used AdamW, with a base learning rate of 1×10^-3, a weight decay coefficient of 1×10^-4, a learning rate scheduler using the cosine annealing warp restarts strategy, and a batch size of 4.

[0197] The principle of this embodiment is as follows: First, the raw multispectral images collected by a rice paddy drone (UAV) are preprocessed. After preprocessing, four types of characteristic spectral bands are integrated to reconstruct a complete image input tensor containing seven bands. A spectral-spatial Transformer backbone network is used as the pre-stage feature extractor to fully model the multi-band spectral information and spatial features. A physical information-guided joint attention module is used as the core of physical-data dual-driven attention enhancement, fusing spectral mutual information, vegetation index physical priors, and spatial-channel attention. The final output is a physical information-enhanced feature. The image is fed into the Unet standard semantic segmentation decoder and then upsampled using stepwise bilinear interpolation to restore it to the original resolution. , , , , The output results are accumulated after bilinear interpolation upsampling, and finally classified pixel by pixel through a Softmax activation layer to output a high-precision weed prediction mask. A multimodal joint loss function is then constructed. The gradient iteration guides the process, mitigating issues such as uneven sample distribution, blurred crop boundary segmentation, and missed detection of small weeds. Finally, comparative and ablation experiments are conducted to validate the data, thereby achieving accurate segmentation and recognition in complex paddy field scenarios.

[0198] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A semantic segmentation method for weeds in rice paddies based on multispectral UAV images, characterized in that, Includes the following steps: S1. Use drones to collect raw multispectral images of rice paddies and perform preprocessing. S2, Multispectral Input and Multiscale Feature Extraction; S3, Physical Information-Guided Joint Attention Module PGMMA Fine-grained Execution; S4. Constraints on the multimodal joint loss function during backpropagation; S5. Comparative and ablation experiments; In step S3, the Physical Information Guided Joint Attention Module (PGMMA) completes adaptive weighted optimization driven by the collaboration of physical rules and network mechanisms during forward propagation. It adopts a dual-input architecture, with one input receiving the high-level spatial semantic feature map output from the SSTB backbone network. ~ , as a data-driven feature stream; Another approach directly inputs the original multispectral image tensor. Using real remote sensing spectral information as the source of physical prior constraints; Step S3 specifically includes the following steps: S3-1, Global Scene Deviation Spatial Attention (GSDA) Submodule The Global Scene Deviation Spatial Attention (GSDA) submodule mines heterogeneous target regions based on global feature statistical differences. It quantifies the degree of feature deviation between pixels and the overall scene through cosine deviation, accurately highlighting the outlier features of weeds. Global average pooling is used to aggregate the spatial information of the entire image to obtain a semantic center vector representing the overall distribution of the rice paddy scene. ； In the formula, H=128 and W=128 are the feature map space dimensions, and the mean feature of this semantic center vector is biased towards the main cultivated rice feature distribution. Using the global mean as a benchmark, the feature vector similarity deviation is calculated pixel by pixel, and the deviation value represents the degree of pixel heterogeneity: ； A deviation value close to 0 indicates that the pixel features fit the mainstream distribution of cultivated rice; a deviation value close to 2 indicates that the pixel is a sparse outlier target that deviates from the overall scene, corresponding to the weedy area in the field. The bias features are learned and the weights are scaled using 1×1 pointwise convolution, and then normalized to the effective weight range using Sigmoid activation. The output generates a global bias space attention map. ； S3-2, Vegetation Index Enhancement Attention VIGA Submodule The vegetation index-enhanced attention VIGA submodule explicitly extracts near-infrared band features corresponding to the backbone network. Features of the red-edge band A lightweight two-stage downsampling module is constructed, consisting of two layers of 3×3 depthwise separable convolutions with a stride of 2, BN and GELU activation functions. Under the premise of low parameter quantity, high-resolution physical features are uniformly mapped to a 128×128 scale, and the aligned physical feature stream is output. Based on the remote sensing physical characteristics of the vegetation red edge effect, and relying on the aligned original physical features, an end-to-end differentiable implicit vegetation index operator is constructed: ； ； In the formula, , The network is designed to learn adaptive scalar parameters, with an initial value of 1.

0. These parameters can dynamically correct residual errors in radiometric calibration of UAV images and spectral nonlinear deviations caused by environmental interference during backpropagation. The Sigmoid activation function is used to normalize the attention weight range. This is a pointwise convolution used to integrate local pixel differences and optimize weight distribution. S3-3, High-efficiency channel-spatial concatenated attention EC-SA submodule The efficient channel-space concatenated attention EC-SA submodule optimizes feature representation from both global channel and local pixel space dimensions, capturing refined geometric features of sparse weeds in paddy fields with extremely low parameter count; the refined geometric features are then fed into the EC-SA submodule for unidirectional concatenated attention enhancement. S3-4, the three-multiplication fusion mechanism of the right-side convergence node The right-side convergence node integrates spectral optimization features, channel spatial enhancement features, and physical prior constraint features using a multi-weighted cascaded product fusion method, achieving multi-dimensional feature adaptive weighted fusion. The weighted fusion formula is as follows: ； In the formula, This is a high-level semantic feature map output by the backbone network; Channel attention weight map; SA is a spatial attention weight map; A vegetation physical attention map driven by the original image; For global bias attention.

2. The semantic segmentation method for rice paddy weeds based on multispectral UAV images according to claim 1, characterized in that, In step S1, the preprocessing step includes: S1-1, sequentially complete radiometric calibration to correct image radiometric errors, vignetting correction to correct lens edge brightness attenuation, noise removal to filter out salt and pepper, streaks, and noise, and geometric registration to correct image distortion and offset. S1-2 integrates the three bands of RGB visible light with four types of characteristic spectral bands: multispectral narrow band red light, multispectral narrow band green light, near-infrared band, and red edge band, and reconstructs a complete image input tensor containing seven bands; S1-3. The fixed input size of the image input tensor is: H×W×C, 512×512×7, where 512 are the image height and width dimensions respectively, and 7 is the number of image bands.

3. The semantic segmentation method for rice paddy weeds based on multispectral UAV images according to claim 2, characterized in that, In step S2, the image input tensor preprocessed in step S1 is input into the SSTB backbone network for independent feature extraction and global modeling. Specific steps include: S2-1, Band Independent Convolutional Encoder: For each band b, b ∈ {1, 2, ..., B}, B=7, local feature extraction is performed using depthwise convolutional groups: , ∈ R^(H×W×C); in: For the input of band b, DWConv is an encoding module containing 3 depthwise separable convolutional layers and batch normalization and GELU activation functions, with C=64 output channels; S2-2, Multi-band feature stitching and fusion: All independent features of all bands are spliced ​​along the channel dimension to form a fused feature tensor: F_concat = Concat[F1, F2, ..., F7] ∈ R^(H×W×(C×7)); Cross-band channel compression is performed using 1×1 convolution, with an output dimension of R^(H×W×D) and D=256. S2-3, Spectral-Spatial Transformer encoder: The fused feature maps are flattened into a sequence and input into the encoder for progressive processing of the fused convolutional features, generating multi-scale feature maps. The encoder consists of 12 Spectral-Spatial Transformer layers, each containing: A. Multi-head self-attention: 6 attention heads are used to capture long-range dependencies between bands; B. Feedforward network: Two-layer MLP with a scaling ratio of 4, used for feature transformation; C. Learnable spectral position coding: Independently learns position coding for each band to distinguish band semantics; D. Residual connectivity and layer normalization: ensure the stability of feature propagation.

4. The semantic segmentation method for rice paddy weeds based on multispectral UAV images according to claim 3, characterized in that, In steps S2-3, the encoder distributes the 12 Transformer layers evenly across four stages, with each stage containing 3 stacked Transformer layers: Stage 1 Shallow Features: The input features undergo preliminary processing, including 4x downsampling, to output a shallow feature map with a size of 128×128×256. ; Stage 2 shallow features: The downsampled features from Stage 1 are processed by an 8-fold downsampling, resulting in a shallow feature map with a size of 64×64×256. ; Stage 3 Mid-to-Deep Features: The Stage 2 downsampled features are processed by a 16x downsampling factor, resulting in a mid-to-deep feature map with a size of 32×32×256. ; Stage 4 Deep Features: The Stage3 downsampled features are processed and downsampled by 32 times to output a deep feature map with a size of 16×16×256. .

5. A semantic segmentation method for rice paddy weeds based on multispectral UAV images according to claim 4, characterized in that, In step S3-3, the execution steps of the unidirectional serial attention enhancement include: S3-31. First, through the channel attention mechanism, relying on the cascaded global max pooling and average pooling combined with a lightweight MLP layer, a 1×1×256-dimensional channel attention weight map CA is generated to complete the suppression of redundant features in the channel dimension and the enhancement of effective features. S3-32. The optimized features are concatenated and fed into the spatial attention mechanism. A 7×7 large receptive field convolutional layer captures the geometric features of a large range of connected regions, generating a 128×128×1 dimension spatial attention weight map SA, which realizes the local spatial region feature focusing and localization.

6. A semantic segmentation method for weeds in rice paddies based on multispectral UAV images according to claim 5, characterized in that, Step S4 specifically includes the following steps: S4-1. Constructing a multimodal joint loss function Guided gradient iteration: ； S4-2, Weighted Multimodal ； In the formula: This represents the foreground prediction probability corresponding to pixel i. The corresponding binary label; Set pixel weight rules: the foreground pixel weight for weeds is 3.0, and the background pixel weight is 1.0; It is a smoothing constant; S4-3 Boundary Distance Transformation Loss ； In the formula: N is the total number of pixels in the image; To predict the pixel features of the mask boundary; It is a distance feature map obtained by transforming the real label mask with Euclidean distance and normalizing it; S4-3, Increased Loss from Small Targets For connected components in an image whose pixel projection area is less than a preset threshold τ (τ=256 pixels), we define them as a sparse set of small targets S, and apply a focus loss constraint to them individually: ； Among them, the preferred value of the focus factor is... , |S| represents the total number of pixels within the small target set.

7. A semantic segmentation method for rice paddy weeds based on multispectral UAV images according to claim 6, characterized in that, In step S5, the comparative experiment includes: S5-1. Perform precise pixel-by-pixel annotation by hand, and divide the training set, validation set, and test set according to a ratio of 7:1.5:1.5; S5-2. Conduct quantitative comparative experiments with mainstream baseline methods on the exact same hardware environment and test set; The ablation experiment involves gradually disassembling or assembling core sub-modules while maintaining complete consistency of the backbone network.

8. A semantic segmentation method for rice paddy weeds based on multispectral UAV images according to claim 7, characterized in that, The hardware environment used is PyTorch 2.0, CUDA 11.8; the network optimizer is AdamW, the base learning rate is set to 1×10^-3, the weight decay coefficient is 1×10^-4, the learning rate scheduler uses cosine annealing, and the batch size is configured to 4.

Images