A sugarcane flowering period recognition method and system based on multi-modal fusion

Abstract

This invention relates to the field of agricultural intelligent identification technology, and in particular to a method for identifying sugarcane flowering period based on multimodal fusion. The method includes the following steps: collecting multimodal data of sugarcane fields, including visible light, near-infrared, and multispectral remote sensing images, meteorological time series data, and field metadata, and obtaining manually labeled flowering stages; preprocessing the multimodal data, including geometric correction and illumination normalization of the remote sensing images, standardization and establishment of a unified time index for the time series data, and interpolation or masking of missing modalities or missing time steps to obtain standardized data. In this invention, by collecting, extracting features from, and aligning across time and space of multimodal data from sugarcane fields, a unified feature vector is generated for flowering period prediction. This improves upon the problem of low accuracy in traditional sugarcane flowering period identification methods that rely on manual visual inspection due to the lack of multimodal fusion and temporal modeling.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for identifying sugarcane flowering period based on multimodal fusion. Background Technology

[0002] With the development of agricultural modernization, intelligent monitoring of crop growth status has become an important component of precision agriculture. Sugarcane, as a major economic crop, directly impacts pollination management, variety selection, and sugar production processes through its flowering stage. Traditional methods for identifying sugarcane flowering stages rely on single remote sensing images, which suffer from low accuracy due to a lack of multimodal fusion and temporal modeling. Therefore, developing an intelligent system capable of fully integrating multimodal data, achieving cross-temporal feature alignment, and accurately identifying sugarcane flowering stages is of great significance for improving the scientific rigor and automation level of large-scale sugarcane planting management. Summary of the Invention

[0003] To overcome the above shortcomings, this invention provides a method and system for sugarcane flowering period identification based on multimodal fusion, aiming to improve the problem of low accuracy in traditional sugarcane flowering period identification methods that use a single remote sensing image due to the lack of multimodal fusion and temporal modeling.

[0004] In a first aspect, the present invention provides the following technical solution: a method for identifying sugarcane flowering period based on multimodal fusion, comprising the following steps:

[0005] S1. Collect multimodal data of sugarcane fields, including visible light, near-infrared and multispectral remote sensing images, meteorological time series data and field metadata, and obtain a small amount of manually labeled flowering stages;

[0006] S2. Preprocess the multimodal data, including geometric correction and illumination normalization of remote sensing images, standardization of time series data and establishment of a unified time index, and interpolation or masking of missing modal or missing time step data to obtain standardized data.

[0007] S3. Modal features are extracted from the standardized data respectively, and modal embedding vectors are generated for each plot at each time step. At the same time, time-series encoding is performed based on the embedding vectors, and time-aware embeddings are generated by combining self-attention mechanism to capture short-term fluctuations and long-term trends.

[0008] S4. Perform cross-temporal self-supervised feature alignment based on the time-aware embedding, including constructing near-temporal cross-modal positive pairs for comparative learning, applying temporal consistency constraints, learning conditional center vectors and mask reconstruction training for land parcels, and obtaining aligned feature embeddings.

[0009] S5. The aligned feature embedding is fused using weighted concatenation or attention mechanism to generate a unified feature vector;

[0010] S6. Train a classifier based on the unified feature vector and predict the flowering stage, outputting the flowering stage information of the plot at each time step.

[0011] By adopting the above technical solution, multimodal data of sugarcane fields are collected, features are extracted, and cross-temporal alignment is performed to generate a unified feature vector for flowering period prediction. This improves the problem of low accuracy in traditional sugarcane flowering period identification methods that use single remote sensing images due to the lack of multimodal fusion and temporal modeling.

[0012] Preferably, the multimodal data includes:

[0013] Geographically locate and divide the fields into regions, generating boundary information and a unique identifier for each field;

[0014] Use drones or satellite equipment to acquire visible light, near-infrared and multispectral remote sensing images, and add time stamps and geographic coordinate information to the images;

[0015] Collect meteorological time series data, including temperature, humidity, precipitation, light intensity and wind speed, and synchronize them in time;

[0016] Collect field metadata, including sugarcane variety, fertilization information, planting density, and soil type;

[0017] A small number of manually labeled flowering stages were collected.

[0018] Preferably, the preprocessing includes:

[0019] Geometric correction and illumination normalization are performed on the acquired remote sensing images;

[0020] Standardize meteorological time series data and establish a unified time index;

[0021] Data with missing modalities or missing time steps are processed by interpolation, mean imputation, or masking to generate a standardized multimodal dataset;

[0022] Perform quality checks on the processed data and remove noisy or abnormal samples.

[0023] Preferably, the modal feature extraction and temporal coding include:

[0024] Use convolutional neural networks to extract modal space features from images;

[0025] Use long short-term memory networks or one-dimensional convolutions to extract meteorological time series features;

[0026] Image modal features and time series features are combined to generate an embedding vector for each plot at each time step;

[0027] The embedding vector is temporally encoded and combined with a self-attention mechanism to generate a time-aware embedding.

[0028] Preferably, the comparative learning includes:

[0029] Construct positive pairs of different modes of the same plot at close time steps, and generate a set of negative samples;

[0030] Perform intermodal comparison learning based on positive and negative samples;

[0031] The sample weights are dynamically adjusted during the comparative learning process.

[0032] Preferably, the timing consistency constraint includes:

[0033] Apply consistency constraints to time series embeddings;

[0034] Constraint embeddings evolve over time;

[0035] Dynamically adjust the consistency constraint weights.

[0036] Preferably, the land parcel learning conditionalization and mask reconstruction include:

[0037] For each plot, a conditional center vector is learned, and the embedding vector is constrained to be close to the center vector;

[0038] Masking is applied to certain time steps or modes;

[0039] The masked embedding is reconstructed using a decoder;

[0040] During training, the center vector is iteratively optimized and the network parameters are reconstructed.

[0041] Preferably, the feature fusion includes:

[0042] The embedded vectors are then weighted and concatenated or fused using an attention mechanism.

[0043] Modal weights and time step importance are considered during the fusion process;

[0044] Generate a unified feature vector.

[0045] Preferably, the flowering stage prediction includes:

[0046] The classifier was fine-tuned using a small number of manually labeled samples;

[0047] Semi-supervised training is performed using high-confidence pseudo-labels;

[0048] The trained classifier is used to predict the flowering stage of each plot at each time step;

[0049] The prediction results are smoothed.

[0050] Secondly, the present invention provides the following technical solution: a sugarcane flowering period identification system based on multimodal fusion, comprising the following modules:

[0051] The data acquisition module is used to collect multimodal data of sugarcane fields, including visible light, near-infrared and multispectral remote sensing images, meteorological time series data and field metadata, and to obtain a small amount of manually labeled flowering stages;

[0052] The data preprocessing module is used to preprocess the multimodal data, including geometric correction and illumination normalization of remote sensing images, standardization and establishment of a unified time index for time series data, and interpolation or masking of missing modal or missing time step data to obtain standardized data.

[0053] The modal feature extraction and temporal coding module is used to extract modal features from the standardized data and generate modal embedding vectors for each plot at each time step. At the same time, temporal coding is performed based on the embedding vectors, and a time-aware embedding is generated by combining a self-attention mechanism to capture short-term fluctuations and long-term trends.

[0054] The cross-temporal self-supervised feature alignment module is used to perform cross-temporal self-supervised feature alignment based on the time-aware embedding, including constructing near-temporal cross-modal positive pairing samples for comparative learning, applying temporal consistency constraints, learning conditional center vectors and mask reconstruction training for land parcels, and obtaining aligned feature embeddings.

[0055] The feature fusion module is used to fuse the aligned feature embeddings through weighted concatenation or attention mechanism to generate a unified feature vector;

[0056] The flowering period identification module is used to train a classifier based on the unified feature vector and predict the flowering period stage, outputting the flowering period stage information of the plot at each time step.

[0057] The present invention has the following beneficial effects:

[0058] 1. In this invention, multimodal data of sugarcane fields are collected, features are extracted, and cross-temporal alignment is performed to generate a unified feature vector for flowering period prediction. This improves the problem of low accuracy in traditional sugarcane flowering period identification methods that use a single remote sensing image due to the lack of multimodal fusion and temporal modeling.

[0059] 2. In this invention, by standardizing remote sensing images and meteorological time series data and interpolating missing data, the consistency of data from different modalities and time steps is ensured. This improves the problem that traditional sugarcane flowering period identification methods mostly use multi-source data that has not been uniformly processed. Due to scale and time differences between data, the model input is incomplete and the training is unstable.

[0060] 3. In this invention, by constructing a time-aware embedding and combining it with cross-temporal self-supervised feature alignment, the feature unification of short-term fluctuations and long-term trends is achieved. This improves the problem that traditional sugarcane flowering period identification methods mostly use single-step feature judgment, which, due to the lack of time continuity constraints, result in discontinuous and significant fluctuations in flowering period prediction results.

[0061] 4. In this invention, by weighted fusion or attention mechanism fusion of aligned feature embeddings, a unified feature vector is generated for classifier training and flowering stage prediction. This improves the problem that traditional sugarcane flowering stage identification methods mostly use single modality or simple splicing, which do not make full use of the differences in the importance of features of each modality, resulting in insufficient model generalization ability. Attached Figure Description

[0062] Figure 1 This is a flowchart of a sugarcane flowering period identification method based on multimodal fusion proposed in this invention.

[0063] Figure 2 This is a flowchart of the feature alignment and fusion technology of a sugarcane flowering period identification method based on multimodal fusion proposed in this invention;

[0064] Figure 3 This is a module architecture diagram of a sugarcane flowering period recognition system based on multimodal fusion proposed in this invention. Detailed Implementation

[0065] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1:

[0067] In a first embodiment of the present invention, the present invention provides a method for sugarcane flowering period identification based on multimodal fusion, such as... Figures 1-2 As shown, it includes the following steps:

[0068] S1. Collect multimodal data of sugarcane fields, including visible light, near-infrared and multispectral remote sensing images, meteorological time series data and field metadata, and obtain a small amount of manually labeled flowering stages;

[0069] Furthermore, multimodal data includes:

[0070] Geographically locate and divide the fields into regions, generating boundary information and a unique identifier for each field;

[0071] Use drones or satellite equipment to acquire visible light, near-infrared and multispectral remote sensing images, and add time stamps and geographic coordinate information to the images;

[0072] Collect meteorological time series data, including temperature, humidity, precipitation, light intensity and wind speed, and synchronize them in time;

[0073] Collect field metadata, including sugarcane variety, fertilization information, planting density, and soil type;

[0074] A small number of manually labeled flowering stages were collected.

[0075] Specifically, the process involves the geographic location and division of target fields. Geographic coordinates are determined for each field using GPS or high-precision surveying techniques to obtain its boundary coordinates. Each field is then divided into independent plot units, and a unique identifier ID is generated for each plot. This identifier ID is denoted as... ,in For the field number, The output data includes the set of land parcel boundary coordinates. and unique identifier .

[0076] Remote sensing image acquisition involves using drones or satellite equipment to collect images of a land parcel, including visible light images. Near-infrared imaging and multispectral images ,in For the time of collection, This refers to the multispectral band number. Each image is timestamped. and geographic coordinate information The output is a standardized image dataset. .

[0077] Meteorological time series data collection involves collecting meteorological data, including temperature, for the location of the plot. ,humidity Precipitation Light intensity and wind speed Data by time step Synchronous recording to generate meteorological time series The output is a time series matrix. .

[0078] Field metadata collection records basic information about the land plots, including sugarcane varieties. Fertilization information Planting density and soil type The data structure is defined as a vector. The output is a set of land parcel metadata. .

[0079] The flowering stages were manually labeled on a small number of plots to create a set of labels. ,in , This indicates the number of categories in the flowering stage. This label is used for subsequent model training and validation. The output is the labeled dataset. .

[0080] Input / output flow; input data includes land parcel coordinate sets. Remote sensing images Meteorological time series and land parcel metadata The output data is a standardized multimodal dataset. The data flow is as follows: .

[0081] Multimodal data mapping is represented as: ;in This represents the data integration function, which includes time alignment, spatial alignment, and label mapping. The formula defines the mapping relationship between the input modal data and the integrated dataset. For the first The collection of plot imagery data includes visible light, near-infrared, and multispectral images; For the first Meteorological time series data for each plot of land; For the first Block metadata; For the first The flowering stage manually marked on each plot of land; For the first Output multimodal data sets for each plot of land.

[0082] After step S1 is completed, the multimodal data and labels of each plot have a unified time step, geographic coordinate information and unique identifier, providing complete input for subsequent preprocessing, feature extraction and flowering period identification.

[0083] S2. Preprocess the multimodal data, including geometric correction and illumination normalization of remote sensing images, standardization of time series data and establishment of a unified time index, and interpolation or masking of missing modal or missing time step data to obtain standardized data.

[0084] Furthermore, the preprocessing includes:

[0085] Geometric correction and illumination normalization are performed on the acquired remote sensing images;

[0086] Standardize meteorological time series data and establish a unified time index;

[0087] Data with missing modalities or missing time steps are processed by interpolation, mean imputation, or masking to generate a standardized multimodal dataset;

[0088] Perform quality checks on the processed data and remove noisy or abnormal samples.

[0089] Specifically, geometric correction of remote sensing images involves performing geometric correction on visible light, near-infrared, and multispectral images to eliminate spatial biases caused by imaging perspective, terrain undulation, and sensor distortion. The correction employs a geographic reference point transformation method to align the image coordinate system with the geographic coordinate system. The geometric correction model is expressed as: ;in These are the original image pixel coordinates. For the corrected pixel coordinates, For geometric transformation functions, based on a set of geographic reference points. Perform linear or polynomial interpolation calculations. The output is a spatially aligned image set. ,in This is a remote sensing image after geometric correction.

[0090] Illumination normalization is performed on the corrected image to reduce the impact of different times and weather conditions on the image brightness distribution. A channel mean-based normalization method is used to standardize the brightness of each spectral channel to a uniform range. The normalization formula is: ;in To correct the image in the band Pixel value at that location, This represents the average pixel value for this band. The standard deviation is given. The output is the illumination-normalized image data. .

[0091] Time series data standardization and time index unification: Meteorological time series data are standardized and a unified time index is established.

[0092] For meteorological sequences The z-score standardization method is used to unify the scale: ;in and These represent the mean and standard deviation of the meteorological data at each time step. A time index set is then established. This ensures that all parcel data is aligned at the same time step. The output is a normalized time series matrix. .

[0093] Missing modalities and missing time steps are addressed by repairing data with missing image modalities or missing time steps. A combination of interpolation, mean imputation, and masking mechanisms is used, with linear interpolation employed. ;in These are the data values ​​for the time steps to be repaired. For completely missing modes, a mask matrix is ​​set. :

[0094] ;

[0095] The output includes a patched multimodal dataset. and the corresponding mask matrix .

[0096] Data quality checks and noise removal: Quality checks are performed on the preprocessed data. Anomaly metrics are calculated for each modality. ;in For sample values, This is the sample mean. If Exceeding the preset threshold If a sample is found to be abnormal, it will be removed. The output is the cleaned and standardized dataset. .

[0097] Input and output flow: Input: raw multimodal data set Output: A standardized and aligned set of multimodal data. .

[0098] The overall processing flow can be represented as follows: ;in This is a preprocessing function for multimodal data, including modules for geometric correction, illumination normalization, time normalization, interpolation, and quality inspection. To output a standardized multimodal data set.

[0099] After this step is completed, all modal data are unified in spatial, illumination, and temporal dimensions, and the data format is consistent, providing a standardized sample set that can be directly input for subsequent feature extraction and cross-modal feature alignment.

[0100] S3. Modal features are extracted from the standardized data respectively, and modal embedding vectors are generated for each plot at each time step. At the same time, temporal encoding is performed based on the embedding vectors, and time-aware embeddings are generated by combining self-attention mechanism to capture short-term fluctuations and long-term trends.

[0101] Furthermore, modal feature extraction and temporal coding include:

[0102] Use convolutional neural networks to extract modal space features from images;

[0103] Use long short-term memory networks or one-dimensional convolutions to extract meteorological time series features;

[0104] Image modal features and time series features are combined to generate an embedding vector for each plot at each time step;

[0105] The embedding vector is temporally encoded and combined with a self-attention mechanism to generate a time-aware embedding.

[0106] Specifically, the standardized remote sensing image is input into a convolutional neural network. Spatial feature information is extracted through multiple convolutional and pooling layers. Convolutional layers are used to extract local texture changes, while pooling layers are used to compress feature dimensions and enhance translation invariance. The output is an image modality feature matrix. This process can be represented as:

[0107] ;

[0108] in Indicates the first Input of remote sensing images at any time This represents the mapping function of a convolutional neural network. For convolutional network parameters, This is the output image modal feature matrix.

[0109] Meteorological time series data are input into a Long Short-Term Memory (LSTM) network for temporal feature extraction, capturing the nonlinear trends and seasonal correlations of meteorological changes. The time series feature extraction process can be represented as follows:

[0110] ;

[0111] in For the first Input of meteorological characteristics at any given time Let be the hidden state vector at the current moment. This refers to the state at the previous moment. This is the set of LSTM parameters.

[0112] Aligning the image modal feature matrix with the hidden state of the meteorological sequence along the time dimension, we form the modal embedding vector for each plot at each time step, defined as: ;in Represents the modal fusion function, output This is the fusion embedding vector for this time step.

[0113] The fused embedded vector sequence is input into a self-attention encoder, and attention weights are calculated between time steps to enhance the ability to capture long-term dependent features. These weights are defined as follows: ;in and The first With the Time step query and key vector, The dimension of the key vector. This represents the attention weights between time steps. The time-aware embedding is obtained by weighted summation:

[0114] ;in For value vectors, For time-aware embeddings that include time context information.

[0115] Input and output flow: Input includes standardized remote sensing image sequences, meteorological time series and their time indices; the feature extraction module generates image modality feature matrices and time series hidden layer features respectively; the fusion module aligns features of different modalities at time steps and generates fused embedding vectors; the self-attention encoding module calculates time-dependent weights based on the fused embedding vectors and generates time-aware embeddings; the output is an embedding sequence containing spatial and temporal correlation features, which is used as input for subsequent flowering period recognition models.

[0116] S4. Perform cross-temporal self-supervised feature alignment based on time-aware embedding, including constructing near-temporal cross-modal positive pairs for comparative learning, applying temporal consistency constraints, learning conditional center vectors and mask reconstruction training for land parcels, and obtaining aligned feature embeddings.

[0117] Furthermore, contrastive learning includes:

[0118] Construct positive pairs of different modes of the same plot at close time steps, and generate a set of negative samples;

[0119] Perform intermodal comparison learning based on positive and negative samples;

[0120] The sample weights are dynamically adjusted during the comparative learning process.

[0121] Timing consistency constraints include:

[0122] Apply consistency constraints to time series embeddings;

[0123] Constraint embeddings evolve over time;

[0124] Dynamically adjust the consistency constraint weights.

[0125] Plot learning conditionalization and mask reconstruction include:

[0126] For each plot, a conditional center vector is learned, and the embedding vector is constrained to be close to the center vector;

[0127] Masking is applied to certain time steps or modes;

[0128] The masked embedding is reconstructed using a decoder;

[0129] During training, the center vector is iteratively optimized and the network parameters are reconstructed.

[0130] Specifically, for each plot's time-aware embedding sequence, different modal features of the same plot at near-term time steps are constructed based on the time index as positive samples, while modal features from other plots or later time phases are selected as the negative sample set. The positive and negative samples are input into the contrastive learning module to calculate the similarity between the embedding vectors, and the loss function is defined as: ;in Embed the vector for the anchor sample. Embed the corresponding positive samples. For negative sample embedding, Represents the cosine similarity function. This represents the temperature coefficient. During training, the weights of positive and negative samples are dynamically adjusted based on the similarity distribution between samples to improve the ability to distinguish feature differences.

[0131] An evolutionary constraint is imposed on the time series embeddings, requiring that the difference in embedding vectors between adjacent time steps be within a controllable range to ensure the continuity of flowering period state changes. The consistency constraint loss is defined as:

[0132] ;

[0133] in and These are the embedding vectors for consecutive time steps. The consistency weights are dynamically adjusted based on the time intervals, reducing the constraint weights for long-interval sample pairs and increasing the weights for short-interval sample pairs, thus reflecting the actual rate of change in flowering period.

[0134] Assign a conditional center vector to each parcel The center vector is updated iteratively through gradient descent, making the embedding vectors closer to the center representation of their respective plots. The clustering constraint loss is defined as:

[0135] ;

[0136] in For the first Embedding vectors of each plot of land, This is the conditional center vector of the land parcel.

[0137] Random masking is applied to certain time steps or modes, with a masking ratio of 1 / 2. The masked embedding vector is input into the reconstruction decoder to generate the reconstructed embedding. And calculate the reconstruction error:

[0138] ;

[0139] By updating the center vector and decoder parameters through backpropagation, the features remain consistent and complete even under conditions of missing modalities or missing time.

[0140] The input and output flow includes time-aware embedding sequences and time indices as inputs; the contrastive learning module constructs positive and negative samples and calculates the intermodal alignment loss; the temporal consistency module constrains the embedding variation amplitude between time steps; the conditional centering module learns the plot center vector and applies clustering constraints; the mask reconstruction module masks some modalities and performs reconstruction training; all loss terms are jointly optimized, and the output is a spatiotemporally aligned feature embedding sequence used as input for the flowering stage discrimination model.

[0141] S5. The aligned feature embeddings are fused using weighted concatenation or attention mechanisms to generate a unified feature vector;

[0142] Furthermore, feature fusion includes:

[0143] Weighted concatenation or attention-based fusion of embedded vectors;

[0144] Modal weights and time step importance are considered during the fusion process;

[0145] Generate a unified feature vector.

[0146] Specifically, for the self-supervised alignment of the multimodal temporal embedding sequence, a weight matrix is ​​first established based on the modality type. And establish a weight matrix based on the importance of time steps. Modal weight matrix ;in For the number of modes, Indicates the first The weight coefficients of each modality in the overall fusion. Time step weight matrix. ,in For the number of time steps, This indicates the weight of that time step.

[0147] Under the weighted splicing and fusion method, the multimodal embedding vector of each plot is... Execute the following weighted combination: ;in The unified feature vector after fusion Representing modes At time step The embedded vectors are weighted and superimposed to represent feature information from different modalities and time phases in a common space.

[0148] When using an attention mechanism for fusion, the input embedding sequence is first processed. Calculate its query, key, and value matrix. : ;in These are the trainable weight matrices. Calculate the attention weight matrix. : ;in For the key vector dimension, the value vector is weighted and summed according to the attention weights to generate the fused output. ;in The fusion feature representation obtained through the attention mechanism can reflect the correlation between different modal features at each time step.

[0149] The input and output flow includes the aligned feature embedding sequence output from step S4 as the input. ; Calculate the modal and time-step weight matrix and If a weighted concatenation method is used, the feature fusion vector is obtained by calculating the weighted sum based on the weight matrix; if an attention mechanism is used, the query, key, and value matrices are calculated and attention weights are generated, and the value matrices are weighted and converged; the output is a unified feature vector. This vector is input into the flowering period classification module for flowering period identification and classification modeling at different stages.

[0150] This step enables the fusion representation of multimodal and temporal features in a unified vector space, ensuring that the input of the downstream classification model has complete cross-modal and temporal correlation information.

[0151] S6. Train a classifier based on a unified feature vector and predict the flowering stage, outputting the flowering stage information of the plot at each time step;

[0152] Furthermore, the flowering stage prediction includes:

[0153] The classifier was fine-tuned using a small number of manually labeled samples;

[0154] Semi-supervised training is performed using high-confidence pseudo-labels;

[0155] The trained classifier is used to predict the flowering stage of each plot at each time step;

[0156] The prediction results are smoothed.

[0157] Specifically, the unified feature vector obtained through step S5 is used. Using the input features, a flowering stage classifier is constructed to determine the flowering stage at different time steps in each plot. The classifier employs a multi-layer fully connected structure or a temporal convolutional network-based architecture to perform linear mapping and nonlinear transformation on the input features.

[0158] The output layer of the classifier uses the Softmax function to calculate the probability distribution for each flowering stage. Let the classifier parameters be... The number of output categories is Then for the input vector The model output probability is:

[0159] ;

[0160] in , For category The weight vector, For the corresponding bias term, Indicates sample Belongs to the Predicted probability of flowering stage.

[0161] The training phase first uses a small number of manually labeled sample sets. The classifier is trained under supervised conditions. The loss function used is cross-entropy loss. ;in For the sample The true label indicator function, when a sample belongs to a category. The value is 1 if it is true, and 0 otherwise.

[0162] To expand the training samples, a semi-supervised training strategy is adopted. First, the current model is used to generate predicted probabilities for unlabeled samples. When the prediction confidence exceeds a threshold... At this point, the predicted labels are added to the training set as pseudo-labels. Let the pseudo-label sample set be... The pseudo-label loss is defined as:

[0163]

[0164] The total loss is the weighted sum of the supervision and false labeling losses: ;in This is a balancing coefficient used to control the impact of pseudo-labeled samples on training.

[0165] The input data during training is a uniform feature vector. The model outputs the predicted probability distribution of the classifier, along with its corresponding flowering period labels (artificial or pseudo-labels). After iterative training, the model parameters are updated. This continues until the loss function converges.

[0166] During the prediction phase, the trained classifier is used to analyze the feature vectors of each plot at each time step. Perform forward inference and output the probability distribution of flowering stage at each time step:

[0167] ;

[0168] The prediction results for each plot are organized according to the time step sequence to form a complete flowering period evolution sequence.

[0169] The prediction results are smoothed by using time-series filtering or exponential smoothing to correct the predicted labels for consecutive time steps. The smoothing formula is as follows: ;in The results are from the smoothed flowering stage. This is the smoothing coefficient, and its value range is... .

[0170] Input and output flow: Input: Unified feature vector from step S5 Output: Flowering stage category for each plot at each time step; Training phase output: Classifier weight parameters; Inference phase output: Predicted flowering stage sequence.

[0171] This step completes the mapping from a multimodal unified feature space to flowering stage discrimination, enabling continuous flowering stage recognition based on time series.

[0172] Example 2:

[0173] In large-scale sugarcane cultivation areas, flowering stage identification is a crucial step in guiding artificial pollination, variety improvement, and production scheduling. Traditional methods rely on manual inspections and analysis of single remote sensing images, which suffer from problems such as discontinuous data acquisition, low identification accuracy, and insufficient real-time performance. To improve the automation level of flowering stage identification, agricultural management departments deploy drones and ground meteorological monitoring stations to regularly collect visible light, near-infrared, and multispectral remote sensing images, and simultaneously acquire meteorological time-series data and field metadata, including sugarcane varieties, planting densities, fertilization information, and soil types. This type of data comes from diverse sources, has inconsistent temporal resolution and spatial scales, and data collected by different sensors exhibits differences in illumination, spatial offsets, and temporal gaps, making it difficult to uniformly model multimodal information. Furthermore, the flowering stage exhibits a clear temporal evolution pattern, but existing identification methods typically only perform static classification on single-phase images, lacking a modeling mechanism for temporal continuity and cross-modal consistency, making it difficult to accurately depict the dynamic process of flowering from its beginning to its peak and decline. Furthermore, multi-source data from large-scale fields often contains missing modalities or anomalous samples during the acquisition process. Without interpolation or masking, noise accumulation can occur, affecting model stability. Simultaneously, the limited number of sugarcane flowering period labeled samples makes it difficult to obtain sufficient supervision signals for model training, resulting in insufficient generalization ability. To address these issues, this invention provides a sugarcane flowering period identification system based on multimodal fusion, the structure of which is as follows: Figure 3 As shown. The specific implementation process of this system is as follows:

[0174] Specifically, the data acquisition module is responsible for acquiring multimodal input data from sugarcane fields, including visible light imagery, near-infrared imagery, multispectral remote sensing imagery, meteorological time-series data, and field metadata. Through multi-source sensing devices and data interfaces, it achieves simultaneous acquisition and structured storage of data from different modalities, forming an input dataset containing spectral, temporal, and geographic attributes, providing complete raw information for subsequent feature extraction and modeling. This module is also responsible for collecting a small amount of manually labeled flowering stage information, which serves as a supervised sample set for the classification model.

[0175] The data preprocessing module standardizes and aligns the acquired raw multimodal data. Geometric correction and illumination normalization are performed on remote sensing imagery to ensure consistent spatial and brightness characteristics across different times and sensing conditions. Meteorological time-series data are standardized, and a unified time index is established for temporal synchronization. Interpolation or masking is performed on data with missing modalities or time steps to form a continuous and complete standardized dataset. This module ensures consistency of input data across spatial, illumination, and temporal dimensions, providing reliable input for subsequent model calculations.

[0176] The modal feature extraction and temporal coding module performs modal feature extraction on the standardized data, generating an embedding vector for each plot at each time step. Image modalities are extracted for spectral features through convolutional or transform coding, while meteorological modalities are extracted for dynamic features through a temporal network. The module uses a time-attention mechanism to weightedly encode features from different time steps, generating time-aware feature vectors that reflect the dynamic patterns of spectral changes and meteorological conditions during the sugarcane flowering period. The module outputs a sequence of embedding vectors containing multimodal time-dependent features.

[0177] The cross-spatiotemporal self-supervised feature alignment module performs cross-spatiotemporal feature alignment based on time-aware embedding. By constructing sample pairs with close time intervals, it performs self-supervised feature learning to ensure that features from adjacent time phases or different modalities remain consistent in a common representation space. The module introduces temporal consistency constraints and mask reconstruction training to correct and complete noise or missing parts in the input features, resulting in aligned feature embeddings with consistent spatiotemporal distribution. The module's technical role is to reduce the differences in feature distribution under different times and sensing conditions, thereby improving the model's stability in cross-temporal prediction.

[0178] The feature fusion module integrates aligned feature embeddings from different modalities to generate a unified feature vector. The fusion process is achieved through weighted concatenation or attention mechanisms, with the weights of different modalities adaptively adjusted based on their feature relevance. The module's output is a unified representation comprehensively reflecting the multidimensional spatiotemporal characteristics of the field, incorporating multi-source information such as spectral, meteorological, and spatial data, providing highly consistent input features for flowering period classification.

[0179] The flowering period identification module trains a classifier based on a unified feature vector and outputs flowering stage information for each plot at each time step. The model is fine-tuned using a small number of manually labeled samples and semi-supervised training is performed using high-confidence pseudo-labels to maintain classification accuracy even with insufficient labeling. After training, the module takes the unified feature vector of the plot as input and outputs the flowering stage category for the corresponding time step. To eliminate short-term noise fluctuations, the system smooths the output results for consecutive time steps to obtain a stable flowering period change sequence. The module's function is to achieve plot flowering period identification and time-series monitoring based on multimodal features, providing classification results for crop growth stage analysis.

[0180] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying sugarcane flowering period based on multimodal fusion, characterized in that, Includes the following steps: S1. Collect multimodal data of sugarcane fields, including visible light, near-infrared and multispectral remote sensing images, meteorological time series data and field metadata, and obtain a small amount of manually labeled flowering stages; S2. Preprocess the multimodal data, including geometric correction and illumination normalization of remote sensing images, standardization of time series data and establishment of a unified time index, and interpolation or masking of missing modal or missing time step data to obtain standardized data. S3. Modal features are extracted from the standardized data respectively, and modal embedding vectors are generated for each plot at each time step. At the same time, time-series encoding is performed based on the embedding vectors, and time-aware embeddings are generated by combining self-attention mechanism to capture short-term fluctuations and long-term trends. S4. Perform cross-temporal self-supervised feature alignment based on the time-aware embedding, including constructing near-temporal cross-modal positive pairs for comparative learning, applying temporal consistency constraints, learning conditional center vectors and mask reconstruction training for land parcels, and obtaining aligned feature embeddings. The conditional center vector for each land parcel is assigned a conditional center vector. The center vector is updated iteratively through gradient descent, making the embedding vectors closer to the center representation of their respective plots. The clustering constraint loss is defined as: ;in For the first Embedding vectors of each plot of land, This is the conditional center vector of the land parcel; S5. The aligned feature embedding is fused using weighted concatenation or attention mechanism to generate a unified feature vector; S6. Train a classifier based on the unified feature vector and predict the flowering stage, outputting the flowering stage information of the plot at each time step.

2. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The multimodal data includes: Geographically locate and divide the fields into regions, generating boundary information and a unique identifier for each field; Use drones or satellite equipment to acquire visible light, near-infrared and multispectral remote sensing images, and add time stamps and geographic coordinate information to the images; Collect meteorological time series data, including temperature, humidity, precipitation, light intensity and wind speed, and synchronize them in time; Collect field metadata, including sugarcane variety, fertilization information, planting density, and soil type; A small number of manually labeled flowering stages were collected.

3. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The preprocessing includes: Geometric correction and illumination normalization are performed on the acquired remote sensing images; Standardize meteorological time series data and establish a unified time index; Data with missing modalities or missing time steps are processed by interpolation, mean imputation, or masking to generate a standardized multimodal dataset; Perform quality checks on the processed data and remove noisy or abnormal samples.

4. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The modality feature extraction and temporal coding include: Use convolutional neural networks to extract modal space features from images; Use long short-term memory networks or one-dimensional convolutions to extract meteorological time series features; Image modal features and time series features are combined to generate an embedding vector for each plot at each time step; The embedding vector is temporally encoded and combined with a self-attention mechanism to generate a time-aware embedding.

5. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The comparative learning includes: Construct positive pairs of different modes of the same plot at close time steps, and generate a set of negative samples; Perform intermodal comparison learning based on positive and negative samples; The sample weights are dynamically adjusted during the comparative learning process.

6. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The timing consistency constraints include: Apply consistency constraints to time series embeddings; Constraint embeddings evolve over time; Dynamically adjust the consistency constraint weights.

7. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The land parcel learning conditionalization and mask reconstruction include: For each plot, a conditional center vector is learned, and the embedding vector is constrained to be close to the center vector; Masking is applied to certain time steps or modes; The masked embedding is reconstructed using a decoder; During training, the center vector is iteratively optimized and the network parameters are reconstructed.

8. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The feature fusion includes: The embedded vectors are then weighted and concatenated or fused using an attention mechanism. Modal weights and time step importance are considered during the fusion process; Generate a unified feature vector.

9. The sugarcane flowering period identification method based on multimodal fusion according to claim 1, characterized in that, The flowering stage prediction includes: The classifier was fine-tuned using a small number of manually labeled samples; Semi-supervised training is performed using high-confidence pseudo-labels; The trained classifier is used to predict the flowering stage of each plot at each time step; The prediction results are smoothed.

10. A sugarcane flowering period recognition system based on multimodal fusion, characterized in that, The sugarcane flowering period identification method based on multimodal fusion, as described in any one of claims 1-9, includes the following modules: The data acquisition module is used to collect multimodal data of sugarcane fields, including visible light, near-infrared and multispectral remote sensing images, meteorological time series data and field metadata, and to obtain a small amount of manually labeled flowering stages; The data preprocessing module is used to preprocess the multimodal data, including geometric correction and illumination normalization of remote sensing images, standardization and establishment of a unified time index for time series data, and interpolation or masking of missing modal or missing time step data to obtain standardized data. The modal feature extraction and temporal coding module is used to extract modal features from the standardized data and generate modal embedding vectors for each plot at each time step. At the same time, temporal coding is performed based on the embedding vectors, and a time-aware embedding is generated by combining a self-attention mechanism to capture short-term fluctuations and long-term trends. The cross-temporal self-supervised feature alignment module is used to perform cross-temporal self-supervised feature alignment based on the time-aware embedding, including constructing near-temporal cross-modal positive pairing samples for comparative learning, applying temporal consistency constraints, learning conditional center vectors and mask reconstruction training for land parcels, and obtaining aligned feature embeddings. The conditional center vector for each land parcel is assigned a conditional center vector. The center vector is updated iteratively through gradient descent, making the embedding vectors closer to the center representation of their respective plots. The clustering constraint loss is defined as: ;in For the first Embedding vectors of each plot of land, This is the conditional center vector of the land parcel; The feature fusion module is used to fuse the aligned feature embeddings through weighted concatenation or attention mechanism to generate a unified feature vector; The flowering period identification module is used to train a classifier based on the unified feature vector and predict the flowering period stage, outputting the flowering period stage information of the plot at each time step.

Images