A rice growth period recognition method and system based on a UAV multi-modal image

Abstract

The application discloses a rice growth period recognition method and system based on a UAV multi-modal image, relates to the technical field of rice growth period recognition, and comprises the following steps: synchronously collecting RGB images and multispectral images of target rice through a UAV platform, and performing pretreatment to generate spatially aligned RGB image blocks and multispectral image blocks; performing feature construction and extraction on the RGB image blocks and the multispectral image blocks to obtain visual depth feature vectors and spectral index feature vectors; constructing a multi-modal fusion network, fusing the visual depth feature vectors and the spectral index feature vectors, and training the multi-modal fusion network by using labeled rice growth period samples to obtain a rice growth period recognition model; and inputting to-be-recognized RGB image blocks and multispectral image blocks into the trained rice growth period recognition model to output corresponding rice growth period recognition results. The application realizes high-precision and strong-robustness recognition of the whole growth period of rice.

Description

Technical Field

[0001] This invention relates to the field of rice growth period identification technology, specifically to a method and system for rice growth period identification based on UAV multimodal images. Background Technology

[0002] Rice, as one of the world's most important food crops, is a key crop for my country's food security. Accurate identification of the rice growth stage is a fundamental step in monitoring crop growth and assessing yield formation, directly impacting refined field management decisions, such as water and fertilizer regulation, pest and disease control, and the timing of key agricultural operations. It also provides crucial prior information for yield prediction and production assessment. Therefore, achieving stable and generalizable growth stage identification in complex field environments is of great significance for improving rice production management efficiency and ensuring food security. Existing methods for monitoring the rice growth stage mainly include traditional manual surveys, machine learning methods based on remote sensing features, and end-to-end deep learning methods.

[0003] Traditional manual survey methods primarily rely on visual assessments or basic measurements by agricultural technicians in the fields. This approach is time-consuming, unsuitable for large-scale monitoring, and highly subjective, with potential biases in judgment among different personnel. Furthermore, its spatial coverage is limited, reflecting only local conditions through sampling points, making high-frequency, large-area automated monitoring difficult. To improve efficiency, existing technologies have further developed machine learning models based on remote sensing features. Researchers use platforms such as drones or satellites to collect crop growth data, including reflectance, vegetation indices, canopy cover, and meteorological data, and then determine and predict crop growth stages or phenological periods. For example, patent application CN116482041B proposes a scheme to rapidly identify the heading stage of rice by extrapolating absorption coefficients from multispectral reflectance, constructing a red-edge correlation index, and combining thresholds with phenological priors. However, these methods are highly dependent on the environment, being very sensitive to light conditions and atmospheric conditions, and their feature engineering often relies on the statistical regularities of specific datasets. For example, patent application CN115759481A inputs remote sensing images and meteorological data (which may include vegetation indices, canopy cover, etc.) into a phenological prediction model to output phenological results and a prediction scheme for the number of days between intervals. However, this type of method has weak generalization ability and is difficult to characterize the continuous physiological evolution of the growth period and the subtle differences between adjacent periods. In recent years, deep learning technology, represented by convolutional neural networks, has made breakthroughs in automatic feature extraction. For example, patent application CN109766957A attempts to use UAV images as input and classify and statistically analyze the growth period of rice through a deep convolutional neural network. It first separates individual plants and extracts their contours before training the CNN for stage recognition. However, this method is significantly affected by field occlusion, dense planting, shadows, and background mixing, and this input has limited expression of physiological information, making it easy to confuse adjacent stages. For example, patent application CN114529097B acquires multi-year remote sensing and phenological data, preprocesses it, and then performs dimensionality reduction to form a variable-length input. An LSTM model is then used to construct a multi-output model, simultaneously identifying the current phenological stage and predicting the next stage. It emphasizes its applicability to regions at different scales. However, this method still needs to address issues such as short time series, irregular sampling, cross-field domain migration, and the fusion of texture and spectral information. Therefore, a fertility period identification technology solution is still needed that can fully utilize the advantages of UAV multimodal observation in complex field environments, improve cross-scene stability, and enhance the separability of adjacent fertility stages. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a method and system for identifying the growth stage of rice based on UAV multimodal images. By constructing a hybrid representation system that combines the deep texture features of RGB images with the interpretable physiological features of multispectral data, it achieves high-precision and robust identification of the entire growth stage of rice.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: a method for identifying the growth stage of rice based on UAV multimodal images, comprising the following steps: S1: Using the multimodal data acquisition and preprocessing module, RGB and multispectral images of the target rice are acquired simultaneously through the UAV platform and preprocessed to generate spatially aligned RGB and multispectral image blocks. S2: Using the multidimensional feature construction and extraction module, features are constructed and extracted from RGB image blocks and multispectral image blocks to obtain visual depth feature vectors and spectral index feature vectors; S3: Using the model training module, a multimodal fusion network is constructed to fuse the visual depth feature vector and the spectral index feature vector. The multimodal fusion network is then trained using labeled rice growth period samples to obtain a rice growth period recognition model. S4: Using the rice growth period recognition module, input the RGB image patch and multispectral image patch to be recognized into the trained rice growth period recognition model, and output the corresponding rice growth period recognition result.

[0006] Furthermore, the multimodal data acquisition and preprocessing module includes: The drone platform flies along a preset route; The RGB imaging submodule is used to acquire high-resolution RGB images; The multispectral imaging submodule is used to acquire four-channel multispectral images, including Green, Red, RedEdge, and NIR bands. The data preprocessing submodule performs the following steps: a1: Import the original RGB image and multispectral image into the measurement software for photogrammetric processing to generate geometrically corrected RGB orthophoto and multispectral orthophoto; a2: Convert the multispectral orthophoto from DN values ​​to surface reflectance, and stack the four single-band reflectance orthophotos in the order of Green, NIR, Red Edge, and Red to form a four-channel multispectral reflectance orthophoto. a3: Perform consistent ROI cropping on RGB orthophoto and multispectral reflectance orthophoto, retaining only the rice canopy region; a4: The cropped image is synchronously segmented using a sliding window to obtain spatially aligned RGB image blocks and multispectral image blocks.

[0007] Furthermore, the multidimensional feature construction and extraction module includes: The depth feature extraction submodule inputs each RGB image block into the ResNet18 convolutional neural network to extract visual depth feature vectors. The vegetation index construction submodule calculates candidate vegetation index maps based on the band reflectance of multispectral image patches, and extracts the mean and standard deviation of each index map to form candidate statistical features. The vegetation index feature screening submodule uses the minimum redundancy maximum correlation algorithm to sort and screen candidate statistical features to obtain spectral index feature vectors.

[0008] Furthermore, the model training module includes: The reproductive period annotation and data segmentation submodule annotates the samples according to the preset reproductive period category system and divides the samples into training set, validation set and test set using orthophoto mosaic image as the basic unit; The multimodal fusion modeling submodule constructs a multimodal fusion network containing RGB branches and exponential branches, fusing visual depth feature vectors and spectral exponential feature vectors; The training optimization and model selection submodule uses labeled rice growth period samples to train the multimodal fusion network, and adopts an early stopping mechanism to select the optimal model weights and hyperparameter configurations to obtain a rice growth period recognition model.

[0009] Furthermore, the multimodal fusion network performs the following sub-steps: b1: The RGB branch uses ResNet18 to extract RGB depth features. After convolutional layers and global average pooling, it outputs an RGB depth feature vector. :

[0010] in, This refers to the ResNet18 network. Represents an RGB image block. Represents the real number field; b2: The exponential branch maps the spectral exponential feature vector to discriminative embedded features through the IndexHead module. :

[0011]

[0012] in, This represents a statistical vector formed by concatenating exponential feature maps in sequence. This represents the exponential feature embedding mapping function. The dimension representing the exponential embedding feature. This represents the mean. Indicates standard deviation, , , Represents the exponential feature graph, with superscript indicating the superscript. Represents the transpose of a matrix; b3: Transfer the RGB depth feature vector With discriminative embedding features The features are then stitched together to obtain the initial fusion features. :

[0013] in, This represents the total dimension of the fused feature after concatenating RGB features and exponential embedding features; b4: Input the initial fused features into the SE attention module to generate a dimension-wise weight vector, and multiply the weight vector and the initial fused features through a channel-wise multiplication to obtain the calibrated fused features. ;

[0014] )) in, This indicates multiplication by channel. Represents a weight vector for each dimension. Represents the ReLU function. This represents the Sigmoid function. and This indicates a fully connected layer.

[0015] Furthermore, the rice growth stage identification module will use the calibrated fused features The high-dimensional features are input into a unified classification head, which maps them to the category space. The softmax activation function is then used to transform the output values ​​into probability distributions of each growth stage category. The category with the highest probability is selected as the rice growth stage identification result.

[0016] The present invention also employs the following technical solution: a rice growth stage recognition system based on UAV multimodal images, comprising: The multimodal data acquisition and preprocessing module is used to simultaneously acquire RGB and multispectral images of the target rice through the UAV platform and perform preprocessing to generate spatially aligned RGB and multispectral image patches. The multidimensional feature construction and extraction module is used to construct and extract features from RGB image patches and multispectral image patches to obtain visual depth feature vectors and spectral index feature vectors. The model training module is used to construct a multimodal fusion network, which fuses visual depth feature vectors and spectral index feature vectors, and trains the multimodal fusion network using labeled rice growth period samples to obtain a rice growth period recognition model. The rice growth period recognition module is used to input the RGB image patch and multispectral image patch to be recognized into the trained rice growth period recognition model and output the corresponding rice growth period recognition result.

[0017] The beneficial effects of this invention are as follows: Compared with traditional manual surveys, threshold / empirical feature-based machine learning methods, and single-modal deep learning methods, the rice growth stage identification method and system based on UAV multimodal images provided by this invention have higher identification accuracy, stability, and generalizability under complex field conditions involving multiple fields, time phases, and light levels. It can effectively alleviate problems such as boundary sample confusion between adjacent growth stages and insufficient generalization across scenarios. Its beneficial effects are mainly reflected in the following aspects: (1) UAV multimodal observation improves information completeness, breaks through the "information blind spot" of single mode, and significantly improves the separability of adjacent reproductive periods. This invention simultaneously introduces RGB high-resolution texture morphology information and four-band multispectral physiological response information (Green, Red, RedEdge, NIR), enabling the model to utilize complementary clues of canopy structure changes and spectral physiological changes at the same time, thereby significantly reducing misjudgment in the transition period of reproductive period and improving the discrimination ability of boundary samples and the overall recognition accuracy; (2) Reflectivity orthorectification, ROI consistency processing, and synchronous patch clipping at the same location significantly reduce the domain shift introduced by differences in illumination and imaging, and improve cross-temporal robustness. The "index graph → statistical phenotypic vector" is used to reduce the high-dimensional sensitivity of multispectral data, and the optimal index subset is formed by mRMR screening to achieve both robustness and interpretability.

[0018] (3) A multimodal deep learning network is used to fuse RGB deep features with Index-Vector phenotypic embedding features, and an SE channel attention mechanism is introduced in the fusion vector layer to perform sample adaptive weighting of the contributions of different modal features. When a certain modality is unstable due to illumination, background or imaging differences, the SE mechanism can suppress the noise channel of that modality and enhance the effective discriminative information of another modality, thereby significantly improving the robustness of the model under complex field perturbations and improving the separability of adjacent stages and the overall stability. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for identifying the growth stage of rice based on UAV multimodal images.

[0020] Figure 2 A flowchart for identifying the growth stages of rice.

[0021] Figure 3 This is a flowchart of the multimodal data acquisition and preprocessing workflow.

[0022] Figure 4 Flowchart for the multidimensional feature construction and extraction module.

[0023] Figure 5 This is a flowchart of the model training module.

[0024] Figure 6 This is a structural diagram of the SE attention module.

[0025] Figure 7 Flowchart for the rice growth stage identification module. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1, such as Figure 1 and Figure 2 As shown, a method for identifying the growth stage of rice based on UAV multimodal images includes the following steps: S1: Using the multimodal data acquisition and preprocessing module, RGB and multispectral images of the target rice are acquired simultaneously through the UAV platform and preprocessed to generate spatially aligned RGB and multispectral image blocks. like Figure 3 As shown, the multimodal data acquisition and preprocessing module includes: The drone platform flies along a preset route; The preferred drone platform is the DJI Mavic 3M aerial survey drone. Under clear or minimally cloudy conditions with an average wind speed not exceeding 1.12 m / s, the drone will fly along a preset route during the stable lighting period from 10:00 to 13:00 local time. The flight altitude is approximately 30 m, the flight path speed is approximately 1.3 m / s, the forward overlap rate is 80%, and the lateral overlap rate is 70%, to ensure that the image coverage and overlap are consistent.

[0028] The RGB imaging submodule is used to acquire high-resolution RGB images to characterize spatial information such as rice canopy texture, row structure and panicle morphology. The multispectral imaging submodule is used to acquire four-channel multispectral images, including Green, Red, RedEdge, and NIR bands, to characterize physiological response information such as changes in canopy chlorophyll, nitrogen, and vitality. The data preprocessing submodule performs the following steps: a1: Import the original RGB and multispectral images into the measurement software (Pix4Dmapper) for photogrammetric processing to generate geometrically corrected RGB and multispectral orthophotos; a2: Convert the multispectral orthophoto from DN values ​​to surface reflectance, and stack the four single-band reflectance orthophotos in the order of Green, NIR, Red Edge, and Red to form a four-channel multispectral reflectance orthophoto. a3: Perform consistent ROI cropping on RGB orthophotos and multispectral reflectance orthophotos, retaining only the rice canopy region, to achieve spatial co-registration and spatiotemporal registration of multimodal data; a4: The cropped image is synchronously divided into 256×256 patch image blocks using a sliding window. RGB and multispectral images are aligned at the same window position and naming rules to ensure pixel-level alignment. An effective pixel ratio threshold of 0.3 is set to remove invalid patches, resulting in spatially aligned RGB and multispectral image blocks.

[0029] S2: Using the multidimensional feature construction and extraction module, features are constructed and extracted from RGB image blocks and multispectral image blocks to obtain visual depth feature vectors and spectral index feature vectors; The multidimensional feature construction and extraction module constructs and extracts features from the RGB patch and the four-channel multispectral reflectance patch output in step S1 to form a complementary "morphological structure-spectral physiology" representation for reproductive period identification.

[0030] like Figure 4 As shown, the multidimensional feature construction and extraction module includes: The deep feature extraction submodule inputs each RGB image block into a ResNet18 convolutional neural network to extract visual depth feature vectors, which characterize the differences in canopy texture, row structure, and ear morphology. Similarly, multispectral data can also be used as input for the depth branch, but the key to this invention is to further vectorize the multispectral information to reduce the high-dimensional sensitivity of pixel-level reflectance and the risk of cross-temporal distribution drift. The vegetation index construction submodule calculates candidate vegetation index maps (such as Normalized Red Edge Difference Index (NDRE), Normalized Difference Vegetation Index (NDVI), Ratio Vegetation Index (RVI), Green Normalized Difference Vegetation Index (GNDVI), Red Edge Chlorophyll Index (RECI), Visible Light Soil-Adjusted Vegetation Index (OSAVI), Modified Simple Ratio Index (MSR), Soil-Adjusted Vegetation Index (SAVI), Improved Soil-Adjusted Vegetation Index (MSAVI), Two-Band Enhanced Vegetation Index (EVI2), etc.) based on the band reflectance of multispectral image patches. Explicit features are constructed in the form of difference / ratio to partially offset multiplicative changes in illumination and suppress additive contributions from the background. Subsequently, the mean and standard deviation (std) are extracted within each patch for each index map, mapping the pixel-level index map to a fixed-length statistical phenotypic vector z, forming a 20-dimensional candidate explicit feature (10 indices × 2 statistics). The vegetation index feature screening submodule uses the minimum redundancy maximum correlation algorithm to sort and screen candidate statistical features in order to reduce collinearity redundancy among candidate explicit features and avoid overfitting caused by high-dimensional explicit features, thus obtaining the spectral index feature vector. The formula for the candidate explicit feature set is as follows: z=[μ(v1),σ(v1),…,μ(v M ),σ(v M )] Among them, v M Let μ(v) represent the Mth exponent. M ) and σ(v M The mean and standard deviation of pixels within a patch are respectively. In this invention, the number of candidate indices is N=10, resulting in a candidate explicit feature dimension of M=10×2=20. Considering the significant collinearity often existing between different spectral indices and their statistical moments, to reduce redundancy and avoid overfitting caused by high-dimensional explicit features while preserving discriminative information, this invention employs minimum redundancy maximum correlation (mRMR) for feature ranking and selection. mRMR uses mutual information I(f;y) to measure the correlation between feature f and the reproductive period label y, as well as the redundancy between features. The iterative feature selection criterion formula is as follows: score(f) = I(f;y)

[0031] Where S represents the selected feature set, and s represents any feature in the selected feature set S, used to measure the redundancy between candidate features and selected features. mRMR first selects the feature that maximizes I(f;y) as the initial feature, and then selects the feature with the highest score in each iteration to add to set S, thus obtaining a feature ranking from strong to weak. This strategy can suppress redundancy while maximizing the discriminative contribution of features, and improve the compactness and generalization of explicit features. Based on the ranking results of mRMR, this invention adopts a forward feature selection strategy to construct feature subsets of different dimensions (taking the top k features, k=1,…,M). Since there may be a complex nonlinear relationship between vegetation index and crop growth period, this invention uses the XGBoost classifier as the evaluator and performs hierarchical 10-fold cross-validation on the training set to quantify model performance. Finally, key index sets RVI, NDVI, NDRE, GNDVI, RECI and OSAVI are obtained at the index level, and their mean and std together constitute a 12-dimensional Index-Vector for subsequent multimodal fusion.

[0032] S3: Using the model training module, a multimodal fusion network is constructed to fuse the visual depth feature vector and the spectral index feature vector. The multimodal fusion network is then trained using labeled rice growth period samples to obtain a rice growth period recognition model. The model training module is used to construct and train a rice growth period recognition model based on the RGB depth features and 12-dimensional Index-Vector obtained in step S2, so that it can maintain stable accuracy under multiple field, multiple time phase and multiple light disturbance conditions.

[0033] like Figure 5 As shown, the model training module includes: The reproductive period labeling and data segmentation submodule labels the samples according to the preset reproductive period category system (including the greening stage, tillering stage, heading stage, milk stage, and full maturity stage). Using orthophoto mosaic images as the basic unit, the samples are divided into training set, validation set, and test set to ensure that different subsets do not share the same orthophoto mosaic image, thus avoiding data leakage caused by spatial correlation from the source.

[0034] After the above preprocessing and partitioning process, a multimodal remote sensing dataset that can be used for training and evaluation of rice growth period identification models was finally constructed. The number of samples in the training set, validation set and test set were 7361, 1199 and 1067, respectively, and the overall ratio of samples in each subset was close to 8:1:1.

[0035] The multimodal fusion modeling submodule constructs a multimodal fusion network containing RGB branches and exponential branches, fusing visual depth feature vectors and spectral exponential feature vectors; The multimodal fusion network performs the following sub-steps: b1: The RGB branch uses ResNet18 to extract RGB depth features. After convolutional layers and global average pooling, it outputs an RGB depth feature vector. :

[0036] in, This refers to the ResNet18 network. Represents an RGB image block. Represents the real number field; The RGB branch, serving as the first branch of the model, is used to process high-resolution RGB images, aiming to capture the texture, color, and geometric differences of the rice canopy. ResNet18 is used as the backbone network. For the input RGB image patch End-to-end feature extraction is performed. After convolutional layers and global average pooling (GAP), a 512-dimensional deep feature vector is output. ; b2: The exponential branch maps the spectral exponential feature vector to discriminative embedded features through the IndexHead module. : To compensate for the shortcomings of low-dimensional vectors in nonlinear representation and to adapt them to high-dimensional RGB features, this invention designs a unified IndexHead module. IndexHead consists of two multilayer perceptron (MLP) layers (structure 12→d→d, default d=32), with ReLU activation and Dropout regularization between layers. This module maps physically meaningful statistical vectors into compact discriminative embedded features. The formula is shown below:

[0037] The exponential branch, serving as the second branch of the model, is used to process multispectral information. To avoid high-dimensional redundancy and overfitting risks caused by directly using pixel-level reflectance, this invention employs a "knowledge compression" strategy, transforming multispectral information into a low-dimensional, interpretable explicit phenotypic statistical vector, the Index Vector. This is based on the exponential set. For the k-th exponential feature map corresponding to the input sample Extract its spatial mean with standard deviation As descriptors, the statistical descriptors of the six indices are then concatenated sequentially to construct a 12-dimensional Index-Vector. The formula is shown below:

[0038] in, This represents a statistical vector formed by concatenating exponential feature maps in sequence. This represents the index feature embedding mapping function, used to encode the input Index-Vector into a low-dimensional discriminative embedding using IndexHead. The dimension representing the exponential embedding feature. This represents the mean. Indicates standard deviation, , , Represents the exponential feature graph, with superscript indicating the superscript. Represents the transpose of a matrix; b3: Transfer the RGB depth feature vector With discriminative embedding features The features are then stitched together to obtain the initial fusion features. :

[0039] in, This represents the total dimension of the fused feature after concatenating RGB features and exponential embedding features; b4: Input the initial fused features into the SE attention module to generate a dimension-wise weight vector, and multiply the weight vector and the initial fused features through a channel-wise multiplication to obtain the calibrated fused features. ; Given that different samples exhibit varying degrees of dependence on "texture morphology" and "physiological phenotype" under conditions of temporal and illumination variations, this invention introduces a lightweight Squeeze-and-Excitation (SE) attention module into the fusion layer, the structure of which is shown in the attached figure. Figure 6 As shown, the feature dimensions are dynamically recalibrated. Given... The SE module directly uses the global feature vector, which already aggregates spatial information, as a context descriptor and generates a dimension-wise weight vector through a "compression-activation" mapping. The formula is shown below: )) Utilizing fully connected layers Compress the feature dimension to The scaling ratio is r=16, and a nonlinearity is introduced through the ReLU activation function to capture cross-modal channel correlations. This is achieved through a second fully connected layer. The dimensions are restored, and a channel-wise weight vector is generated using the Sigmoid function. .

[0040] Final calibrated fusion features The formula can be obtained by multiplying through each channel as follows:

[0041] in, This indicates multiplication by channel. Represents a weight vector for each dimension. Represents the ReLU function. This represents the Sigmoid function. and This indicates a fully connected layer.

[0042] Then input a unified classification header and output the probability of the reproductive period category.

[0043] The training optimization and model selection submodule uses labeled rice growth period samples to train the multimodal fusion network. The loss function can be cross-entropy loss or its improved form. The optimization strategy is consistent with the learning rate scheduling, and the early stopping mechanism is used to select the optimal model weights and hyperparameter configuration to obtain the rice growth period identification model.

[0044] S4: Using the rice growth period recognition module, input the RGB image patch and multispectral image patch to be recognized into the trained rice growth period recognition model, and output the corresponding rice growth period recognition result.

[0045] like Figure 7 As shown, the rice growth stage identification module will use the calibrated fusion features. The input is fed into a unified classification head, which consists of Dropout layers and fully connected layers. This head maps high-dimensional features to a class space and uses the Softmax activation function to transform the output values ​​into probability distributions belonging to each reproductive stage category.

[0046] The corresponding categories are, in order, greening, tillering, booting, heading, milk stage, and full maturity. The category with the highest probability is taken as the growth stage identification result for that patch, i.e., Result = argmax(P). Next, this invention performs a statistical evaluation of the rice growth status within the monitoring area based on the identification results. After the system completes the identification of all patches for a certain plot, it automatically generates a growth stage distribution map for that plot and sends the generated identification report to the user terminal in text or visual chart form.

[0047] Example 2: A rice growth stage recognition system based on UAV multimodal images, comprising: The multimodal data acquisition and preprocessing module is used to simultaneously acquire RGB and multispectral images of the target rice through the UAV platform and perform preprocessing to generate spatially aligned RGB and multispectral image patches. The multidimensional feature construction and extraction module is used to construct and extract features from RGB image patches and multispectral image patches to obtain visual depth feature vectors and spectral index feature vectors. The model training module is used to construct a multimodal fusion network, which fuses visual depth feature vectors and spectral index feature vectors, and trains the multimodal fusion network using labeled rice growth period samples to obtain a rice growth period recognition model. The rice growth period recognition module is used to input the RGB image patch and multispectral image patch to be recognized into the trained rice growth period recognition model and output the corresponding rice growth period recognition result.

[0048] In summary, this invention constructs a dual-stream fusion network framework for rice RGB and multispectral data. It extracts spatial morphological information with RGB deep features as the backbone, while introducing multispectral information as a supplement. Through multimodal complementarity, it effectively eliminates the information blind spots of a single modality at a specific growth stage.

[0049] This invention proposes an explicit phenotypic vectorization method based on "knowledge compression." Instead of directly using unstable pixel-level multispectral reflectance, this invention selects key vegetation indices such as NDRE, NDVI, and RVI based on agronomic mechanisms, and extracts their statistical features (mean and standard deviation) to construct a low-dimensional index feature vector. This "knowledge compression" strategy significantly reduces data dimensionality and environmental noise interference while preserving physiological spectral information, thus significantly improving the model's generalization ability across temporal and site scenarios.

[0050] This invention introduces an attention mechanism to achieve adaptive feature recalibration, incorporating a lightweight Squeeze-and-Excitation (SE) attention module into the feature fusion layer. This module can adaptively learn and adjust the weights of RGB texture features and exponential spectral features based on the quality of the current input sample, acting as a "modal soft switch." This allows it to automatically strengthen the most advantageous discriminative features under interference conditions such as changes in illumination or texture blurring, further improving the accuracy and stability of recognition.

[0051] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.

Claims

1. A method for identifying the growth stage of rice based on UAV multimodal images, characterized in that, Includes the following steps: S1: Using the multimodal data acquisition and preprocessing module, RGB and multispectral images of the target rice are acquired simultaneously through the UAV platform and preprocessed to generate spatially aligned RGB and multispectral image blocks. S2: Using the multidimensional feature construction and extraction module, features are constructed and extracted from RGB image blocks and multispectral image blocks to obtain visual depth feature vectors and spectral index feature vectors; S3: Using the model training module, a multimodal fusion network is constructed to fuse the visual depth feature vector and the spectral index feature vector. The multimodal fusion network is then trained using labeled rice growth period samples to obtain a rice growth period recognition model. The multimodal fusion network performs the following sub-steps: b1: The RGB branch uses ResNet18 to extract RGB depth features. After convolutional layers and global average pooling, it outputs an RGB depth feature vector. : in, This refers to the ResNet18 network. Represents an RGB image block. Represents the real number field; b2: The exponential branch maps the spectral exponential feature vector to discriminative embedded features through the IndexHead module. : in, This represents a statistical vector formed by concatenating exponential feature maps in sequence. This represents the exponential feature embedding mapping function. This indicates the dimension of the discriminant embedded features. This represents the mean. Indicates standard deviation, , , Represents the exponential feature graph, with superscript indicating the superscript. Represents the transpose of a matrix; b3: Transfer the RGB depth feature vector With discriminative embedding features The features are then stitched together to obtain the initial fusion features. : in, This represents the total dimension of the fused feature after concatenating the RGB depth features and the discriminative embedded features; b4: Input the initial fused features into the SE attention module to generate a dimension-wise weight vector, and multiply the weight vector and the initial fused features through a channel-wise multiplication to obtain the calibrated fused features. ; in, This indicates multiplication by channel. Represents a weight vector for each dimension. Represents the ReLU function. This represents the Sigmoid function. and Indicates a fully connected layer; S4: Using the rice growth period recognition module, input the RGB image patch and multispectral image patch to be recognized into the trained rice growth period recognition model, and output the corresponding rice growth period recognition result.

2. The method for identifying the rice growth stage based on UAV multimodal images according to claim 1, characterized in that, The multimodal data acquisition and preprocessing module includes: The drone platform flies along a preset route; The RGB imaging submodule is used to acquire high-resolution RGB images; The multispectral imaging submodule is used to acquire four-channel multispectral images, including Green, Red, RedEdge, and NIR bands. The data preprocessing submodule performs the following steps: a1: Import the original RGB image and multispectral image into the measurement software for photogrammetric processing to generate geometrically corrected RGB orthophoto and multispectral orthophoto; a2: Convert the multispectral orthophoto from DN values ​​to surface reflectance, and stack the four single-band reflectance orthophotos in the order of Green, NIR, Red Edge, and Red to form a four-channel multispectral reflectance orthophoto. a3: Perform consistent ROI cropping on RGB orthophoto and multispectral reflectance orthophoto, retaining only the rice canopy region; a4: The cropped image is synchronously segmented using a sliding window to obtain spatially aligned RGB image blocks and multispectral image blocks.

3. The method for identifying the rice growth stage based on UAV multimodal images according to claim 1, characterized in that, The multidimensional feature construction and extraction module includes: The depth feature extraction submodule inputs each RGB image block into the ResNet18 convolutional neural network to extract visual depth feature vectors. The vegetation index construction submodule calculates candidate vegetation index maps based on the band reflectance of multispectral image patches, and extracts the mean and standard deviation of each index map to form candidate statistical features. The vegetation index feature screening submodule uses the minimum redundancy maximum correlation algorithm to sort and screen candidate statistical features to obtain spectral index feature vectors.

4. The method for identifying the rice growth stage based on UAV multimodal images according to claim 1, characterized in that, The model training module includes: The reproductive period annotation and data segmentation submodule annotates the samples according to the preset reproductive period category system and divides the samples into training set, validation set and test set using orthophoto mosaic image as the basic unit; The multimodal fusion modeling submodule constructs a multimodal fusion network containing RGB branches and exponential branches, fusing visual depth feature vectors and spectral exponential feature vectors; The training optimization and model selection submodule uses labeled rice growth period samples to train the multimodal fusion network, and adopts an early stopping mechanism to select the optimal model weights and hyperparameter configurations to obtain a rice growth period recognition model.

5. The method for identifying the rice growth stage based on UAV multimodal images according to claim 4, characterized in that, The rice growth stage identification module will use the calibrated fusion features The high-dimensional features are input into a unified classification head, which maps them to the category space. The softmax activation function is then used to transform the output values ​​into probability distributions of each growth stage category. The category with the highest probability is selected as the rice growth stage identification result.

6. A system for rice growth stage identification based on UAV multimodal images as described in any one of claims 1-5, characterized in that, include: The multimodal data acquisition and preprocessing module is used to simultaneously acquire RGB and multispectral images of the target rice through the UAV platform and perform preprocessing to generate spatially aligned RGB and multispectral image patches. The multidimensional feature construction and extraction module is used to construct and extract features from RGB image patches and multispectral image patches to obtain visual depth feature vectors and spectral index feature vectors. The model training module is used to construct a multimodal fusion network, which fuses visual depth feature vectors and spectral index feature vectors, and trains the multimodal fusion network using labeled rice growth period samples to obtain a rice growth period recognition model. The rice growth period recognition module is used to input the RGB image patch and multispectral image patch to be recognized into the trained rice growth period recognition model and output the corresponding rice growth period recognition result.

Images