Multi-crop multi-modal recognition method based on deep learning

Abstract

The application discloses a multi-crop multi-modal recognition method based on deep learning, and relates to the technical field of agricultural image recognition. The method comprises the following steps: collecting and labeling original field data sets with crop categories and growth stage labels; dividing the data sets and constructing an initial model containing visible light, thermal infrared and multi-spectral convolution branches; respectively extracting deep feature maps of each mode; recalibrating feature weights through a cross-modal attention fusion layer to generate a joint feature tensor that fuses texture and physiological state; inputting the tensor into a time series modeling unit for dynamic modeling of the growth cycle to output a space-time correlation feature vector; finally, completing category and growth stage recognition through a classification head, and optimizing model parameters using labels until convergence. The application realizes fine fusion of multi-modal information and dynamic modeling of crop growth time series, and improves the accuracy and robustness of multi-crop recognition and growth stage monitoring.

Description

Technical Field

[0001] This invention belongs to the field of agricultural image recognition technology, specifically a multi-crop multimodal recognition method based on deep learning. Background Technology

[0002] Accurate identification and growth monitoring of field crops are core requirements for smart agriculture management. Existing technologies mainly rely on single visible light images, using convolutional neural networks for feature extraction and classification. These methods are sensitive to crop surface texture and color features, but are easily affected by changes in light intensity and foliage shading, resulting in insufficient stability. To address the limitations of single-modality methods, multimodal recognition methods combining visible light, multispectral, or thermal infrared images have emerged. Existing fusion techniques often employ feature concatenation or weighted summation. These strategies treat all modal features equally, failing to differentiate and integrate heterogeneous information according to specific tasks, leading to feature redundancy and the suppression of crucial physiological information. For example, physiological state information such as crop water stress and temperature differences contained in thermal infrared modalities are weighted similarly to the texture and edge information of visible light during simple superposition, weakening the ability to identify crop health status. Furthermore, existing models are mostly static recognition frameworks, treating crop growth as an instantaneous state and lacking modeling of the continuous dynamic changes throughout the crop growth cycle. This results in poor model adaptability to phenotypic differences at different growth stages of the same crop, making it difficult to accurately delineate key phenological periods such as heading, flowering, and grain filling, thus limiting its application value in precision agricultural decision-making. Furthermore, existing monitoring schemes based on optical remote sensing satellites are significantly constrained by weather conditions such as clouds, fog, and rainfall, easily leading to data gaps during key phenological stages and interruptions in growth monitoring. This invention aims to address the problem of refined synergistic fusion among multimodal features and the effective modeling of dynamic information on crop growth timelines. Summary of the Invention

[0003] This invention aims to solve at least one of the technical problems existing in the prior art; To this end, the present invention proposes a multi-crop multimodal recognition method based on deep learning, comprising: Collect raw field crop datasets and label each sample in the raw field crop dataset with crop category labels and growth stage labels; The original field crop dataset is divided into a training subset and a validation subset, and the training subset is used to construct an initial multimodal deep learning model containing visible light convolution branches, thermal infrared convolution branches, and multispectral convolution branches. The training subset is input into the visible light convolution branch, the thermal infrared convolution branch, and the multispectral convolution branch respectively for feature extraction to obtain the visible light deep feature map, the thermal infrared deep feature map, and the spectral deep feature map. The visible light deep feature map, thermal infrared deep feature map, and spectral deep feature map are fed into the cross-modal attention fusion layer for weight recalibration to generate a joint feature tensor that integrates texture information and physiological state. The joint feature tensor is input into the temporal modeling unit to perform dynamic modeling of the growth cycle, and outputs a spatiotemporal correlation feature vector with a time dimension. The spatiotemporal correlation feature vector is fed into the classification decision head to determine the crop category and growth stage, generating a preliminary prediction result matrix. The preliminary prediction result matrix is ​​compared with the crop category label and growth stage label to calculate the backpropagation gradient and update the parameters of the initial multimodal deep learning model until the model converges, thus obtaining the trained deep learning model.

[0004] Further, the original field crop dataset is divided into a training subset and a validation subset, and an initial multimodal deep learning model is constructed using the training subset, including: The original field crop dataset includes visible light images, thermal infrared images, and multispectral data. A stratified sampling strategy is adopted. According to the sample proportions of different crop categories and different growth stages in the original field crop dataset, samples are randomly drawn from each category-stage combination according to a preset proportion in order to maintain the original data distribution and construct the training subset and validation subset. For the visible light convolutional branch, an encoder structure based on stacked depthwise separable convolutional blocks is constructed, wherein each depthwise separable convolutional block contains a depthwise convolutional layer and a pointwise convolutional layer, used to extract spatial texture features from the visible light image; For the thermal infrared convolution branch, an encoder structure including a hollow spatial pyramid pooling module is constructed. The hollow spatial pyramid pooling module is configured with multiple convolutional layers with different void ratios in parallel to capture multi-scale thermal radiation features from thermal infrared images. For the multispectral convolution branch, a band feature encoder containing a one-dimensional convolutional layer is constructed. The one-dimensional convolutional layer performs convolution operations in the spectral dimension to extract spectral reflectance features from the multispectral band data. The outputs of the visible light convolutional branch, the thermal infrared convolutional branch, and the multispectral convolutional branch are connected through a shared cross-modal feature alignment layer. The cross-modal feature alignment layer maps the feature maps output by the three branches to a unified feature dimension through a learnable linear projection matrix, thereby forming the initial multimodal deep learning model.

[0005] Furthermore, after dividing the original field crop dataset into training and validation subsets, a data cleaning step is also included: Iterate through each sample in the training subset and extract the sharpness score of the visible light image and the signal-to-noise ratio of the thermal infrared image of the sample. Set a sharpness score threshold and a signal-to-noise ratio (SNR) index threshold, and mark samples whose sharpness score is lower than the sharpness score threshold or whose SNR index is lower than the SNR index threshold as low-quality samples; All low-quality samples in the training subset are removed, and the missing positions after removal are filled with neighboring samples using an interpolation algorithm to ensure the data integrity of the training subset. The cleaned training subset is then input back into the initial multimodal deep learning model for subsequent training.

[0006] Further, the training subset is input into the visible light convolution branch, the thermal infrared convolution branch, and the multispectral convolution branch respectively for feature extraction, including: For the visible light convolutional branch, a depth-separable convolutional layer is used to replace the standard convolutional layer, and spatial feature dimensionality reduction is performed on the visible light images in the training subset to obtain low-resolution visible light spatial features. For the aforementioned thermal infrared convolution branch, a hollow spatial pyramid pooling structure is introduced to extract the physiological features of the multi-scale receptive field from the thermal infrared images in the training subset, thereby obtaining the thermal features characterizing the crop transpiration state. For the multispectral convolution branch, the differences between the multispectral bands in the training subset are calculated to generate a vegetation index feature map, and the vegetation index feature map is convolutionally encoded to obtain biochemical component features. The visible light spatial features, thermal features, and biochemical component features are normalized and adjusted to a uniform tensor size before being output.

[0007] Furthermore, the visible light deep feature map, thermal infrared deep feature map, and spectral deep feature map are fed into a cross-modal attention fusion layer for weight recalibration, generating a joint feature tensor that integrates texture information and physiological state, including: In the cross-modal attention fusion layer, a query matrix, a key matrix, and a value matrix are constructed, wherein the query matrix is ​​generated by the visible light deep feature map, and the key matrix is ​​formed by splicing the thermal infrared deep feature map and the spectral deep feature map; Calculate the relevance score between the query matrix and the key matrix, and generate an attention weight distribution map based on the relevance score; The value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation of enhanced physiological state. The enhanced feature representation of the physiological state is concatenated with the visible light deep feature map through channels, and the channel dimension is compressed through convolution operation to generate the joint feature tensor.

[0008] Furthermore, the value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation of enhanced physiological state, including: Extract the multimodal attention weight vector corresponding to the thermal infrared deep feature map and the spectral deep feature map from the attention weight distribution map; The multimodal attention weight vector is multiplied element-wise with the eigenvalues ​​at the corresponding positions in the value matrix to obtain the weighted thermal infrared feature blocks and spectral feature blocks. The weighted thermal infrared feature blocks and spectral feature blocks are spliced ​​along the channel dimension to form a temporary enhanced feature set; A cross-channel weighted average calculation is performed on the features in the enhanced feature set to generate a physiological state enhanced feature representation that reflects the intensity of crop transpiration and nutrient absorption status.

[0009] Furthermore, the joint feature tensor is input into the temporal modeling unit for dynamic modeling of the growth cycle, outputting a spatiotemporal correlation feature vector with a time dimension, including: The joint feature tensor is reshaped into a time-series sample sequence, where each element of the time-series sample sequence corresponds to the crop feature state at a specific acquisition time. The time-series sample sequence consists of multiple data points from the same field location at different collection times. Each data point contains visible light, thermal infrared, and multispectral information for that location at the corresponding time. The time-series sample sequence is input into a bidirectional gated loop unit for forward and backward information transmission to capture the continuous change pattern of crop growth. At each time step, the hidden state of the bidirectional gated loop unit is extracted and residually connected with the original joint feature tensor corresponding to the time step. Global max pooling is performed on the features processed at all time steps to aggregate them and obtain the spatiotemporal correlation feature vector with time dimension.

[0010] Further, the preliminary prediction result matrix is ​​compared with the crop category label and growth stage label to calculate the backpropagation gradient and update the parameters of the initial multimodal deep learning model, including: The crop category label is converted into a real category vector in one-hot encoded form, and the growth stage label is converted into a serialized growth stage vector. The preliminary prediction result matrix is ​​split into crop category prediction vectors and growth stage prediction vectors; Calculate the cross-entropy loss between the crop category prediction vector and the true category vector, and the mean squared error loss between the growth stage prediction vector and the growth stage vector, respectively. The cross-entropy loss value and the mean squared error loss value are weighted and summed according to a preset scaling factor to obtain the composite loss function value. The partial derivatives of all weights in the initial multimodal deep learning model are calculated based on the composite loss function value, and the weight parameters are updated using an adaptive moment estimator optimizer.

[0011] Furthermore, after obtaining the trained deep learning model, a model lightweighting step is also included: Analyze the parameter distribution of each branch in the trained deep learning model and identify the top few convolutional layers with the highest parameter proportions. The output channels of the first few convolutional layers are evaluated using a channel pruning algorithm, and redundant channels with a contribution lower than the pruning threshold are removed. The pruned network structure is reconstructed by using fully connected layers, replacing the original fully connected method with a grouped convolutional connection method. Export the pruned and reconstructed network structure and weight files, replace the original trained deep learning model, and obtain a lightweight inference model suitable for embedded devices.

[0012] Furthermore, the value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation of enhanced physiological state, including: Extract the first modal attention weight vector corresponding to the thermal infrared deep feature map and the second modal attention weight vector corresponding to the spectral deep feature map from the attention weight distribution map; The first modality attention weight vector is multiplied element-wise with the feature values ​​from the thermal infrared deep feature map in the value matrix to obtain the weighted thermal infrared feature block; The second modality attention weight vector is multiplied element-wise with the feature values ​​from the deep spectral feature map in the value matrix to obtain the weighted spectral feature blocks; The weighted thermal infrared feature blocks and the weighted spectral feature blocks are spliced ​​along the channel dimension to form an enhanced feature set; A global average pooling operation is performed on the enhanced feature set along the channel dimension to generate a single-channel global feature representation that reflects the physiological state of the crop. The single-channel global feature representation is extended along the spatial dimension and fused with the weighted thermal infrared feature blocks and the weighted spectral feature blocks by dot product to generate the final physiological state enhanced feature representation.

[0013] Compared with the prior art, the beneficial effects of the present invention are: By recalibrating the weights of visible light, thermal infrared, and multispectral deep features through a cross-modal attention fusion layer, this mechanism can dynamically evaluate the importance of each modal feature at different spatial locations and in different channels to the current recognition task. For textured edge regions in the visible light feature map, the attention mechanism assigns higher weights to enhance morphological information; for temperature anomaly regions in the thermal infrared feature map related to crop transpiration and moisture status, the mechanism automatically enhances their response. This ensures that the generated joint feature tensor is no longer a simple superposition of the original features, but a discriminative expression that deeply integrates phenotypic texture and intrinsic physiological state information. This enables the model to make more robust recognitions based on differences in crop intrinsic physiological activities in complex field environments, reducing misjudgments caused by relying solely on appearance changes.

[0014] A temporal modeling unit is introduced to dynamically model the growth cycle of the joint feature tensor. This unit can learn and memorize the feature evolution patterns during the growth process. It establishes a temporal correlation between static, discrete field sampling point features, and the output spatiotemporally correlated feature vector not only contains the phenotypic and physiological information at the current moment, but also implicitly implies the logical continuity with previous growth states. This design enables the model to understand the continuous process of crop growth from vegetative to reproductive growth, thus classifying growth stages based on the dynamic evolution trajectory of features, rather than just instantaneous snapshots. The model therefore possesses the ability to continuously perceive and infer the crop growth process, and its growth stage classification results are more consistent with the continuity laws in agronomical terms. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of the deep learning-based multi-crop multimodal recognition method described in this invention. Figure 2 The flowchart for cleaning the training subset data.

[0016] Figure 3 For timing modeling, see the unit structure diagram; Figure 4 Visualization of multimodal data and deep features; Figure 5 This is a detailed structural diagram of the cross-modal attention fusion layer. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention provides a deep learning-based multi-crop multimodal recognition method, which includes the following steps: (See attached diagram) Figure 1 A raw field crop dataset was collected, and each sample in the dataset was labeled with a crop category label and a growth stage label. The raw field crop dataset was divided into a training subset and a validation subset. An initial multimodal deep learning model containing visible light convolutional branches, thermal infrared convolutional branches, and multispectral convolutional branches was constructed using the training subset. The training subset was then input into the visible light convolutional branch, thermal infrared convolutional branch, and multispectral convolutional branch respectively for feature extraction, resulting in visible light deep feature maps, thermal infrared deep feature maps, and spectral deep feature maps. These feature maps were then fed into a cross-modal attention fusion layer for weight recalibration, generating a joint feature tensor that integrates texture information and physiological state. This joint feature tensor was input into a temporal modeling unit for dynamic modeling of the growth cycle, outputting a spatiotemporal correlation feature vector with a time dimension. This spatiotemporal correlation feature vector was then fed into a classification decision head for crop category determination and growth stage segmentation, generating a preliminary prediction result matrix. The preliminary prediction result matrix is ​​compared with the crop category label and growth stage label to calculate the backpropagation gradient and update the parameters of the initial multimodal deep learning model until the model converges, thus obtaining the trained deep learning model.

[0019] In one embodiment of the present invention, the original field crop dataset includes visible light images, thermal infrared images, and multispectral band data. A stratified sampling strategy is employed, randomly sampling samples from each category-stage combination according to a preset proportion based on the sample ratio of different crop categories and different growth stages in the original field crop dataset, thus maintaining the original data distribution and constructing training and validation subsets. For the visible light convolutional branch, an encoder structure based on stacked depthwise separable convolutional blocks is constructed, where each depthwise separable convolutional block contains a depthwise convolutional layer and a pointwise convolutional layer, used to extract spatial texture features from the visible light images. For the thermal infrared convolutional branch, an encoder structure including a dilated spatial pyramid pooling module is constructed, where the dilated spatial pyramid pooling module sets multiple convolutional layers with different dilation rates in parallel, used to capture multi-scale thermal radiation features from the thermal infrared images. For the multispectral convolutional branch, a band feature encoder including a one-dimensional convolutional layer is constructed, where the one-dimensional convolutional layer performs convolution operations in the spectral dimension, used to extract spectral reflectance features from the multispectral band data. The outputs of the visible light convolutional branch, the thermal infrared convolutional branch, and the multispectral convolutional branch are connected through a shared cross-modal feature alignment layer. The cross-modal feature alignment layer maps the feature maps output by the three branches to a unified feature dimension through a learnable linear projection matrix, so as to form an initial multimodal deep learning model.

[0020] In practice, the original field crop dataset includes visible light images, thermal infrared images, and multispectral data collected by synchronous sensing devices. The visible light images are RGB three-channel color images with a resolution of 1920×1080 pixels. The thermal infrared images are grayscale images representing temperature distribution. The multispectral data includes reflectance values ​​for five specific bands: red, green, blue, red-edge, and near-infrared. For each sample unit in the dataset—a combination of visible light images, thermal infrared images, and multispectral data collected at the same location and time—crop category and growth stage labels are added based on agronomical expert knowledge. Crop category labels include corn, wheat, and rice. Growth stage labels are divided into seedling stage, jointing stage, heading stage, and maturity stage based on the main phenological stages. The method is universally applicable to data acquisition platforms. When applied to UAVs or aerial platforms, the spatial resolution, spectral band range, and radiometric calibration parameters of the visible light images, thermal infrared images, and multispectral data need to be adjusted according to the characteristics of the selected airborne sensor to ensure that the input data meets the processing requirements of each convolutional branch. The multi-temporal data required for the time series modeling unit should be collected according to the aerial survey plan of the UAV or airborne platform.

[0021] In some embodiments, a stratified sampling strategy is employed to construct training and validation subsets. Specifically, the stratified sampling strategy involves statistically analyzing the number and proportion of samples from different crop categories and growth stages in the original field crop dataset. The original field crop dataset is acquired through satellite remote sensing platforms. Visible light images are sourced from high-resolution optical satellites, thermal infrared images from satellites equipped with thermal infrared sensors, and multispectral band data from multispectral satellites. Satellite remote sensing platforms include, but are not limited to, Sentinel-2, Landsat series, Gaofen series, or Planet satellite constellations. By periodically imaging the same area, large-scale, multi-temporal remote sensing images with spatiotemporal consistency are obtained to construct a dataset suitable for macro-scale crop identification and monitoring. For example, an original dataset may contain a total of 12,000 samples, with 5,000 samples for the crop category "maize," 4,000 for "wheat," and 3,000 for "rice." Within each crop category, the sample distribution across the four growth stages may differ. The stratified sampling strategy randomly selects samples from each "category-stage" combination according to a predetermined proportion, based on the sample ratios of different crop categories and growth stages in the original field crop dataset. The predetermined split ratio is 80% for the training subset and 20% for the validation subset. For the "maize-seedling" combination, if there are 1000 samples in the original data, 800 are randomly selected for the training subset and 200 for the validation subset. This process is repeated for all "category-stage" combinations, resulting in training and validation subsets that maintain consistency with the joint distribution of crop category and growth stage in the original field crop dataset. The formula for calculating the sample size in the stratified sampling strategy can be expressed as: in, Representative from the first Crop category, number The number of samples drawn into the training subset from each growth stage combination. This represents the total number of samples for this combination in the original field crop dataset. Represents the preset training set ratio, symbol This indicates the floor function. Correspondingly, it represents the number of samples in the validation subset. .

[0022] In the specific implementation, for the visible light convolutional branch, an encoder structure based on stacked depthwise separable convolutional blocks is constructed. The encoder structure contains four sequentially connected depthwise separable convolutional blocks. Each depthwise separable convolutional block contains a depthwise convolutional layer and a pointwise convolutional layer. The depthwise convolutional layer uses a 3×3 kernel, and the pointwise convolutional layer uses a 1×1 kernel. Each depthwise separable convolutional block is followed by a batch normalization layer and a ReLU activation function. The number of output channels of the four depthwise separable convolutional blocks are set to 32, 64, 128, and 256, respectively. After the first and third depthwise separable convolutional blocks, a 2×2 max-pooling layer with a stride of 2 is inserted to downsample the spatial dimension of the visible light image, thereby extracting hierarchical spatial texture features.

[0023] In the specific implementation, an encoder structure including a dilated spatial pyramid pooling module is constructed for the thermal infrared convolutional branch. The front end of the encoder structure consists of two standard convolutional layers for initial extraction of shallow features from the thermal infrared image. The dilated spatial pyramid pooling module sets four convolutional layers with different dilation rates in parallel, namely 1, 6, 12, and 18. Each convolutional layer uses a 3×3 kernel and has the same number of output channels. The dilated spatial pyramid pooling module performs convolution operations with four different dilation rates in parallel on the input shallow features, capturing multi-scale thermal radiation features with different spatial receptive fields. The output of the dilated spatial pyramid pooling module is obtained by concatenating the results of the four parallel paths along the channel dimension.

[0024] In the specific implementation, a band feature encoder containing one-dimensional convolutional layers is constructed for the multispectral convolutional branch. The input multispectral band data is organized into a one-dimensional vector of length 5, where each vector element represents the reflectance value of a band. The one-dimensional convolutional layer performs convolution operations along the spectral dimension, using a one-dimensional convolutional kernel of width 3 that slides along the spectral dimension. The band feature encoder consists of two stacked one-dimensional convolutional layers: the first maps the input channels from 5 to 16, and the second maps the number of channels from 16 to 32. Each one-dimensional convolutional layer is followed by batch normalization and a ReLU activation function, ultimately outputting the spectral reflectance features extracted from the multispectral band data.

[0025] In practice, the outputs of the visible light convolutional branch, the thermal infrared convolutional branch, and the multispectral convolutional branch are connected through a shared cross-modal feature alignment layer. This layer receives feature maps from the three branches: the visible light convolutional branch has a size of H×W×256, the thermal infrared convolutional branch has a size of H×W×C1, and the multispectral convolutional branch is spatially expanded to H×W×C2. The cross-modal feature alignment layer maps the feature maps from the three branches to a unified feature dimension D using three learnable linear projection matrices. Specifically, each projection matrix linearly transforms the feature vector at each spatial location of the input feature map. After processing by the cross-modal feature alignment layer, the output feature maps of the three branches become H×W×D in both spatial size and channel dimension. These feature maps collectively constitute the initial multimodal feature representation of the multimodal deep learning model, which is then used by subsequent fusion and processing modules.

[0026] In one embodiment of the present invention, after dividing the original field crop dataset into a training subset and a validation subset, a data cleaning step is performed. (See also...) Figure 2 The algorithm iterates through each sample in the training subset, extracting the sharpness score of the visible light image and the signal-to-noise ratio (SNR) of the thermal infrared image. Sharpness score and SNR thresholds are set, and samples with either a sharpness score or SNR below the threshold are marked as low-quality samples. All low-quality samples in the training subset are removed, and interpolation algorithms are used to fill the gaps with neighboring samples to ensure data integrity. The cleaned training subset is then input back into the initial multimodal deep learning model for subsequent training. For the visible light convolutional branch, depthwise separable convolutional layers are used instead of standard convolutional layers to reduce the spatial features of the visible light images in the training subset, resulting in low-resolution visible light spatial features. For the thermal infrared convolutional branch, a hollow spatial pyramid pooling structure is introduced to extract physiological features of the multi-scale receptive field from the thermal infrared images in the training subset, obtaining thermal features characterizing crop transpiration. For the multispectral convolution branch, the differences between multispectral bands in the training subset are calculated to generate a vegetation index feature map. This feature map is then convolved and encoded to obtain biochemical component features. The visible light spatial features, thermal features, and biochemical component features are then normalized and adjusted to a uniform tensor size before being output.

[0027] In practice, after dividing the original field crop dataset into training and validation subsets, a data cleaning step is performed. The data cleaning step involves iterating through each sample in the training subset and extracting the sharpness score of the visible light image and the signal-to-noise ratio (SNR) of the thermal infrared image. The sharpness score is calculated based on the Laplacian operator response of the visible light image; after applying a Laplacian filter to the image, its variance is calculated, with a larger variance indicating a sharper image. The SNR of the thermal infrared image is obtained by calculating the ratio of the signal mean of the uniform crop region to the standard deviation of the background noise region. Sharpness score thresholds and SNR thresholds are set, with the sharpness score threshold set to a specific value and the SNR threshold set to another specific value. Samples with sharpness scores below the sharpness score threshold or SNR thresholds below the SNR threshold are marked as low-quality samples. The formula for calculating the sharpness score can be expressed as: in, Represents the clarity score. The function represents the input visible light RGB image. This indicates that the Laplace filter operator is used for convolution, and the function... This indicates the calculation of the variance of all elements in a matrix. (Clarity rating) The larger the value, the better the image. The higher the resolution, the better.

[0028] In some embodiments, all low-quality samples in the training subset are removed, and an interpolation algorithm is used to fill the gaps with neighboring samples. The interpolation algorithm employs a K-nearest neighbor-based imputation method. For each removed low-quality sample's position in the feature space, the K nearest neighbors are found in the remaining samples of the training subset. A new synthetic sample is generated using the weighted average of the feature vectors of the visible light image, thermal infrared image, and multispectral band data of these K neighboring samples to fill the gaps left by the removed samples. The interpolation algorithm ensures the data integrity of the training subset in terms of sample quantity, preventing subsequent model training from being interrupted due to data loss. After data cleaning and imputation, the cleaned training subset is re-input into the initial multimodal deep learning model for subsequent training processes, including feature extraction, feature fusion, and model optimization.

[0029] In the specific implementation, for the visible light convolution branch, a depthwise separable convolutional layer is used instead of a standard convolutional layer to reduce the spatial features of the visible light images in the training subset. The depthwise separable convolutional layer consists of two sequential parts: depthwise convolution and pointwise convolution. The depthwise convolution performs independent spatial convolution on each channel of the input visible light image, while the pointwise convolution mixes the information from each channel using a 1×1 convolutional kernel. By stacking multiple depthwise separable convolutional layers with a stride of 2, the spatial size of the visible light image is gradually reduced, ultimately yielding low-resolution visible light spatial features. This visible light spatial feature is a three-dimensional tensor with an increased number of channels but significantly reduced height and width, thus achieving the compression and encoding of spatial texture information in the original image.

[0030] In some embodiments, a hollow spatial pyramid pooling structure is introduced for the thermal infrared convolutional branch to extract physiological features of the receptive field at multiple scales from the thermal infrared images in the training subset. The hollow spatial pyramid pooling structure comprises four parallel convolutional paths, each using a 3×3 convolutional kernel but with different void ratios set to 1, 6, 12, and 18, respectively. Convolutional layers with larger void ratios can expand the receptive field without increasing the number of parameters, thereby capturing a wider range of temperature distribution contextual information in the thermal infrared images. The outputs of the four paths are concatenated along the channel dimension to form a multi-scale feature map. This feature map integrates thermal radiation patterns from local to global perspectives and is defined as a thermodynamic feature characterizing the crop's transpiration state.

[0031] In practical implementation, for the multispectral convolutional branch, the differences between multispectral bands in the training subset are calculated to generate a vegetation index feature map. The multispectral band data must contain reflectance values ​​for at least the red and near-infrared bands. The vegetation index feature map is generated by calculating the normalized difference vegetation index (NDV), which is calculated as (near-infrared reflectance - red reflectance) divided by (near-infrared reflectance + red reflectance). The generated vegetation index feature map is then convolutionally encoded. The encoder consists of two standard two-dimensional convolutional layers with a 3×3 kernel size, used to extract spatial texture information related to leaf area index and chlorophyll content, ultimately obtaining biochemical component features characterizing the biochemical components of the crop. It can be understood that, in addition to the normalized difference vegetation index, other types of vegetation indices, such as enhanced vegetation indices, can also be calculated.

[0032] In practice, visible light spatial features, thermal features, and biochemical component features are each normalized and adjusted to a uniform tensor size before output. Normalization includes layer normalization, which standardizes the mean and variance of each feature tensor along the channel dimension. Size adjustment is achieved through bilinear interpolation upsampling or max-pooling downsampling operations, ensuring that the feature tensors from the three branches have identical height, width, and number of channels. For example, the visible light spatial feature size is adjusted to H×W×C, and the thermal and biochemical component features are also adjusted to the same H×W×C size through upsampling or projection operations. The adjusted three feature tensors are aligned in both numerical range and spatial dimension, providing standardized input for subsequent cross-modal feature fusion operations. It can be understood that a uniform size is fundamental to achieving effective interaction and computation of different modal features.

[0033] In one embodiment of the present invention, a query matrix, a key matrix, and a value matrix are constructed in the cross-modal attention fusion layer. The query matrix is ​​generated from a visible light deep feature map, and the key matrix is ​​formed by concatenating a thermal infrared deep feature map and a spectral deep feature map. A correlation score between the query matrix and the key matrix is ​​calculated, and an attention weight distribution map is generated based on the correlation score. The value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation for enhanced physiological state. A first-modal attention weight vector corresponding to the thermal infrared deep feature map and a second-modal attention weight vector corresponding to the spectral deep feature map are extracted from the attention weight distribution map. The first-modal attention weight vector is element-wise multiplied with the eigenvalues ​​from the thermal infrared deep feature map in the value matrix to obtain a weighted thermal infrared feature block. The second-modal attention weight vector is element-wise multiplied with the eigenvalues ​​from the spectral deep feature map in the value matrix to obtain a weighted spectral feature block. The weighted thermal infrared feature blocks and the weighted spectral feature blocks are concatenated along the channel dimension to form an enhanced feature set. A global average pooling operation is performed on the enhanced feature set along the channel dimension to generate a single-channel global feature representation reflecting the crop's physiological state. This single-channel global feature representation is then extended along the spatial dimension and fused with weighted thermal infrared feature blocks and weighted spectral feature blocks through dot product to generate the final physiologically enhanced feature representation. This physiologically enhanced feature representation is then concatenated with the visible light deep feature map through channel concatenation, and the channel dimension is compressed using a convolution operation to generate a joint feature tensor.

[0034] In practical implementation, a query matrix, key matrix, and value matrix are constructed in the cross-modal attention fusion layer. The construction of these matrices relies on learnable linear transformation weights. Specifically, the visible light deep feature map, thermal infrared deep feature map, and spectral deep feature map are first concatenated along the channel dimension to form a combined feature tensor. Three independent linear projection operations are applied to this combined feature tensor to generate the query matrix, key matrix, and value matrix, respectively. The query matrix is ​​generated separately from the visible light deep feature map through a linear projection transformation, emphasizing its role as an information retrieval agent. The key matrix is ​​generated by concatenating the thermal infrared deep feature map and the spectral deep feature map along the channel dimension, followed by another linear projection transformation, representing the retrieved information source. The value matrix is ​​typically generated from the same feature concatenation as the key matrix through a third linear projection, used to store the information content to be aggregated.

[0035] In some embodiments, a relevance score between the query matrix and the key matrix is ​​calculated, and an attention weight distribution map is generated based on the relevance score. The relevance score is calculated by matrix multiplication using the transpose of the query matrix and the key matrix. The resulting original relevance score matrix is ​​scaled by a scaling factor equal to the reciprocal of the square root of the key matrix dimension, and then normalized along the corresponding dimension of the key matrix using the softmax function to generate the attention weight distribution map. Each element in the attention weight distribution map represents the weight assigned to each key position when generating the final output. The formula for calculating the attention weight distribution map can be expressed as: in, This represents the generated attention weight distribution map. Represents the query matrix. Represents the bond matrix. Represents the transpose of the key matrix. This represents the feature dimension of each key vector in the key matrix. Operator This represents a normalized exponentiation operation along a specified dimension, such that the sum of the weights in each row is 1. It can be understood that the scaling operation is to prevent scaling along any dimension. When the value is large, the dot product result becomes too large, leading to the vanishing gradient problem of the softmax function.

[0036] In practice, the attention weight distribution map is used to perform a weighted summation operation on the value matrix to generate a feature representation that enhances the physiological state. The first modality attention weight vector corresponding to the thermal infrared deep feature map and the second modality attention weight vector corresponding to the spectral deep feature map are extracted from the attention weight distribution map. The attention weight distribution map corresponds to the source of the input features in the channel dimension, and the weight parts corresponding to different input modalities can be separated through slicing. The first modality attention weight vector is multiplied element-wise with the feature values ​​from the thermal infrared deep feature map in the value matrix to obtain the weighted thermal infrared feature blocks. The second modality attention weight vector is multiplied element-wise with the feature values ​​from the spectral deep feature map in the value matrix to obtain the weighted spectral feature blocks. The element-wise multiplication operation multiplies the feature value at each position with the corresponding attention weight scalar, amplifying high-weight features and suppressing low-weight features.

[0037] In some embodiments, weighted thermal infrared feature blocks and weighted spectral feature blocks are concatenated along the channel dimension to form an enhanced feature set. A global average pooling operation is performed on the enhanced feature set along the channel dimension to generate a single-channel global feature representation reflecting the crop's physiological state. The global average pooling operation averages the enhanced feature set along the spatial dimensions (height and width) to obtain a one-dimensional vector of length equal to the total number of channels, which is then compressed into a single-channel feature map through a fully connected layer. The single-channel global feature representation is expanded along the spatial dimension and fused with the weighted thermal infrared feature blocks and the weighted spectral feature blocks through a dot product to generate the final physiological state-enhanced feature representation. The dot product fusion is performed channel-wise. The single-channel global feature map is first expanded to the same number of channels as the weighted feature blocks by replication, then multiplied element-wise with each of the two weighted feature blocks, and finally the two multiplication results are added. This process allows global physiological state context information to modulate and enhance the multimodal feature response at each spatial location. Optionally, a convolutional layer can be added after the dot product fusion for feature mixing.

[0038] In the specific implementation, the enhanced physiological state feature representation is concatenated with the visible light deep feature map through channels, and the channel dimension is compressed through convolution operations to generate a joint feature tensor. The channel concatenation operation directly connects the channels of the enhanced physiological state feature representation with the channels of the visible light deep feature map, forming an intermediate feature tensor with double the number of channels. Then, a 1×1 convolutional layer is used to convolve this intermediate feature tensor. The role of the 1×1 convolutional layer is cross-channel information fusion and dimensionality reduction. It projects the concatenated high-dimensional features into a pre-defined, lower-dimensional feature space, ultimately outputting a joint feature tensor that integrates visible light texture information, thermal infrared physiological information, and spectral biochemical information. It can be understood that the 1×1 convolutional layer effectively achieves deep fusion and compression of cross-modal features without changing the size of the feature map space.

[0039] See Figure 3 This is a structural diagram of the temporal modeling unit, which elaborates on the internal working mechanism of the core module of the invention, "Dynamic Modeling of Growth Cycle." It uses multi-temporal fusion features acquired from key phenological nodes such as seedling stage, jointing stage, heading stage, grain-filling stage, and maturity stage of the same plot as sequential input. Two parallel sub-modules, "Bidirectional GRU" and "Self-Attention," work together: the Bidirectional GRU captures the forward and backward dependencies of crop growth states in the temporal dimension, i.e., how the current state is influenced by the past and predicts the future; while the Self-Attention mechanism can mine the global correlation between any two temporal phases in the entire growth sequence, identifying phenological patterns spanning the entire cycle. The outputs of both are fused to form the final high-dimensional "spatiotemporal feature vector." This design directly corresponds to and implements the definition in the specification of "learning the complete spectrum, texture, and continuous temperature change trajectory of crops," representing a key technological achievement in solving the industry problem of "different spectra for the same crop, and the same spectrum for different crops." It significantly improves the stability and accuracy of recognition through dynamic patterns in the temporal dimension.

[0040] In one embodiment of the invention, a multimodal attention weight vector corresponding to the thermal infrared deep feature map and the spectral deep feature map is extracted from the attention weight distribution map. The multimodal attention weight vector is then multiplied element-wise with the corresponding feature values ​​in the value matrix to obtain weighted thermal infrared feature blocks and spectral feature blocks. These weighted thermal infrared and spectral feature blocks are then concatenated along the channel dimension to form a temporary enhanced feature set. A cross-channel weighted average is calculated on the features in the enhanced feature set to generate a physiological state enhanced feature representation reflecting the intensity of crop transpiration and nutrient absorption. The joint feature tensor is reshaped into a time-series sample sequence, where each element corresponds to the crop feature state at a specific acquisition time. The time-series sample sequence is input into a bidirectional gated recurrent unit for forward and backward information transmission, capturing the continuous change patterns of crop growth. In this way, the time-series modeling unit learns and models not static features of a single time phase, but the dynamic evolution trajectory of the crop's texture, thermal radiation, and spectral reflectance from its early stages to maturity. This spatiotemporal correlation feature vector can characterize the complete growth and phenological curve of the crop. At each time step, the hidden state of the bidirectional gated recurrent unit is extracted and residually concatenated with the original joint feature tensor corresponding to the time step. Global max pooling is performed on the features processed at all time steps to aggregate them into a spatiotemporal correlation feature vector with a time dimension.

[0041] In practice, multimodal attention weight vectors corresponding to the thermal infrared deep feature map and the spectral deep feature map are extracted from the attention weight distribution map. The attention weight distribution map is a three-dimensional tensor with dimensions of [batch size, number of query positions, number of key positions]. The extraction of multimodal attention weight vectors is accomplished by slicing along the key position dimension, which corresponds to the source order of the input features. Assuming that the first half of the key matrix corresponds to the thermal infrared deep feature map and the second half corresponds to the spectral deep feature map, the attention weight distribution map is halved along the key position dimension to obtain the first modal attention weight vector corresponding to the thermal infrared deep feature map and the second modal attention weight vector corresponding to the spectral deep feature map. These vectors together constitute the multimodal attention weight vector.

[0042] In some embodiments, element-wise multiplication is performed between the multimodal attention weight vector and the corresponding eigenvalues ​​in the value matrix to obtain weighted thermal infrared feature blocks and spectral feature blocks. The channel dimension organization of the value matrix is ​​consistent with that of the key matrix. For the value matrix, the first part of the channels corresponds to the thermal infrared deep feature map, and the second part of the channels corresponds to the spectral deep feature map. The element-wise multiplication operation involves multiplying the first modal attention weight vector corresponding to each query position element-wise with all channel features in the thermal infrared part of the value matrix, and similarly multiplying the second modal attention weight vector element-wise with all channel features in the spectral part of the value matrix. Table 1 below shows a numerical example of element-wise multiplication between the multimodal attention weight vector extracted at a certain spatial location and the eigenvalues ​​of the value matrix in a simplified scenario: Table 1: Numerical Table of Multimodal Attention Weight Vectors In practice, the weighted thermal infrared feature blocks and spectral feature blocks are spliced ​​along the channel dimension to form a temporary enhanced feature set. The weighted thermal infrared feature blocks have Each channel, the weighted spectral feature block has The temporary enhanced feature set generated by concatenating multiple channels along the channel dimension has ( The system has 10 channels, with spatial dimensions consistent with the input feature map. A cross-channel weighted average is calculated on the features in the enhanced feature set to generate a physiological state enhanced feature representation reflecting the intensity of crop transpiration and nutrient absorption. The cross-channel weighted average calculation is achieved by applying a one-dimensional convolution operation along the channel dimension to the temporary enhanced feature set. The kernel size of this one-dimensional convolution is 1, and the number of input channels is ( The output channel number is 1. This operation can be described as linearly combining the feature maps of all channels and compressing them into a single-channel feature map. The calculation formula is: in, The enhanced physiological state features generated represent the values ​​at a certain spatial location. Represents the first convolution kernel and the second... The learnable weight parameters corresponding to each input channel The first one in the temporary enhanced feature set The feature values ​​of each channel at the same spatial location. This can be understood as learnable weight parameters. During model training, backpropagation is used for optimization to achieve adaptive selection and fusion of important channel features.

[0043] In some embodiments, the joint feature tensor is reshaped into a temporal sample sequence, where each element corresponds to a crop feature state at a specific acquisition time. The original shape of the joint feature tensor is [batch size, number of channels, height, width]. The reshaping operation first merges the batch size dimension and the number of channels dimension, then flattens the height and width dimensions into a spatial location dimension, ultimately resulting in a temporal sample sequence with the shape [time step, number of samples, feature dimension]. The time step T corresponds to the number of samples at different acquisition times of the input model, the number of samples equals the batch size multiplied by the total number of spatial locations (height × width), and the feature dimension equals the number of channels in the joint feature tensor. It can be understood that each "sample" in this context represents a feature vector of a spatial location at a specific time.

[0044] In practice, the time-series sample sequence is input into a bidirectional gated recurrent unit (BRU) for forward and backward information transmission, capturing the continuous variation patterns of crop growth. The BRU consists of a forward recurrent network and a backward recurrent network. The forward recurrent network processes the time-series sample sequence from the first time step to the last time step, while the backward recurrent network processes it from the last time step back to the first time step. At each time step, the BRU updates the hidden state based on the current input and the hidden state of the previous time step. For a time-series sample sequence of length T, the BRU ultimately outputs the forward and backward hidden states for each time step. These hidden states are concatenated along the feature dimension to obtain the context-aware hidden state for each time step. At each time step, the hidden state of the BRU is extracted and residually connected to the original joint feature tensor corresponding to that time step. The original joint feature tensor needs to be reshaped to the same shape as the time-series sample sequence. The residual connection adds the context-aware hidden state of the corresponding time step element-wise to the original feature vector. Optionally, a linear projection layer can be applied to the original feature vectors before addition to align the feature dimensions.

[0045] In practice, global max pooling is performed on the features processed at all time steps to aggregate them into a spatiotemporal correlation feature vector with a time dimension. After residual connection, the output of each time step is a two-dimensional matrix, where rows correspond to samples (spatial locations) and columns correspond to features. The global max pooling operation is performed along the time step dimension, selecting the maximum value for each feature channel of each sample across all T time steps. This operation preserves the most significant feature response for each feature channel at each spatial location throughout the entire growth cycle. The feature vector output by the global max pooling operation no longer contains the time dimension; its shape is [number of samples, feature dimension]. This feature vector is the final output, a spatiotemporal correlation feature vector that integrates spatial features and temporal evolution patterns.

[0046] See Figure 4 This is a visualization of multimodal data and deep features. Through illustrative examples, it provides an intuitive visual reference for understanding the abstract concepts of "multimodality" and "feature fusion." The image presents three original data formats from the same observation unit side-by-side: a "true-color visible light" image reflecting surface texture and structure; a "thermal infrared temperature map" visually displaying surface temperature distribution through pseudo-color mapping, relating it to crop water stress and transpiration; and a "multispectral false-color" composite image highlighting vegetation biochemical attributes (such as chlorophyll content) through band combinations. Below are the "texture feature map" and "thermal anomaly feature map" extracted by the network. These intermediate features visualize the discriminative information under different modalities of interest to the network. Finally, all information is integrated into a "joint feature tensor." This image vividly illustrates the complementarity of multi-source data and how neural networks, through hierarchical abstraction and fusion, comprehensively transform multiple physical observations into a mathematical representation rich in semantics, supporting accurate identification in the end.

[0047] In one embodiment of the present invention, crop category labels are converted into one-hot encoded true category vectors, and growth stage labels are converted into serialized growth stage vectors. The initial prediction result matrix is ​​split into crop category prediction vectors and growth stage prediction vectors. The cross-entropy loss value between the crop category prediction vector and the true category vector, and the mean squared error loss value between the growth stage prediction vector and the growth stage vector are calculated respectively. The cross-entropy loss value and the mean squared error loss value are weighted and summed according to a preset scaling factor to obtain the composite loss function value. The partial derivatives of all weights in the initial multimodal deep learning model are calculated based on the composite loss function value, and the weight parameters are updated using an adaptive moment estimation optimizer. The constructed model can learn and use the complete growth phenological curve of the crop, that is, the trajectory of multimodal features changing over time, for discrimination. This addresses common challenges in single-phase recognition, such as different crops exhibiting similar phenotypic features at specific growth stages, or the same crop exhibiting different phenotypes on the same calendar day due to environmental differences. This invention significantly improves the accuracy and robustness of multi-crop identification and growth stage segmentation by leveraging its temporal modeling capabilities to differentiate based on the dynamic evolution of features over time, rather than relying solely on instantaneous features. After obtaining the trained deep learning model, a model lightweighting step is performed. The parameter distribution of each branch in the trained deep learning model is analyzed, identifying the top convolutional layers with the highest parameter proportions. A channel pruning algorithm is used to evaluate the output channels of the top convolutional layers, removing redundant channels with contributions below the pruning threshold. The pruned network structure is then reconstructed using fully connected layers, replacing the original fully connected layers with grouped convolutional connections. The pruned and reconstructed network structure and weight files are exported, replacing the original trained deep learning model to obtain a lightweight inference model suitable for embedded devices.

[0048] In practice, crop category labels are converted into one-hot encoded true category vectors, and growth stage labels are converted into serialized growth stage vectors. For a batch of samples, the crop category label is a one-dimensional vector, where each element is an integer representing the category, such as 0 for corn, 1 for wheat, and 2 for rice. One-hot encoding expands each integer label into a binary vector of length equal to the total number of categories, setting the value to 1 only at the corresponding category index and 0 at the rest. Growth stage labels are typically floating-point numbers representing consecutive growth stages, such as 0.2 for seedling stage and 0.5 for jointing stage. Serialized growth stage vectors directly organize these floating-point values ​​into vector form without one-hot encoding. It can be understood that growth stage labels can also be encoded as integers based on discrete phenological periods, in which case they also need to be converted into one-hot encoded form.

[0049] In practice, the initial prediction result matrix is ​​split into crop category prediction vectors and growth stage prediction vectors. The initial prediction result matrix output by the classification decision head is a two-dimensional matrix, with the number of rows equal to the number of samples in the batch, and the number of columns equal to the sum of the number of crop categories and the output dimension of the growth stage. Through matrix slicing, the first C columns (C being the total number of crop categories) are extracted. This part, after being processed by the softmax function, represents the probability distribution of each sample belonging to each crop category, and is called the crop category prediction vector. The remaining columns are extracted; this part represents the growth stage prediction value for each sample, and is called the growth stage prediction vector. In some embodiments, if the growth stage is a regression prediction, the growth stage prediction vector is a real-valued vector; if the growth stage is a classification prediction, this part is also processed by a softmax function.

[0050] In practice, the cross-entropy loss between the crop category prediction vector and the true category vector, and the mean squared error loss between the growth stage prediction vector and the growth stage vector are calculated separately. The cross-entropy loss measures the difference between the predicted probability distribution and the true distribution. For each sample, the formula for calculating the cross-entropy loss is: in It is the value (0 or 1) of the true class vector in the i-th class. This represents the predicted probability of the crop category prediction vector in the i-th class. The final classification loss is obtained by averaging the cross-entropy loss values ​​over all samples in the entire batch. The mean squared error loss measures the squared difference between the regression prediction and the true value. The formula for calculating the mean squared error loss for each sample is: in These are the values ​​in the actual growth stage vector. This is the value in the prediction vector during the growth phase. The final regression loss is obtained by averaging over the entire batch.

[0051] In some embodiments, the cross-entropy loss value and the mean squared error loss value are weighted and summed according to a preset scaling factor to obtain the composite loss function value. The preset scaling factor is used to balance the contributions of the classification task and the regression task to the total loss. The formula for calculating the weighted sum is: in, This represents the value of the composite loss function. This represents the mean of the calculated cross-entropy loss values. This represents the mean of the calculated mean squared error loss values. and These are preset non-negative weighting coefficients used to adjust the relative importance of classification loss and regression loss. Optionally, [the following can be added]: and Setting both to 1.0 indicates that the two losses are given equal weight. This can be understood as the weighting coefficient. and The value can be set before model training according to the specific needs of the task.

[0052] In practice, partial derivatives of all weights in the initial multimodal deep learning model are calculated based on the composite loss function value, and the weight parameters are updated using an adaptive moment estimation optimizer. The calculation of partial derivatives is automated via backpropagation, starting from the composite loss function value and proceeding backward along the computation graph, applying the chain rule to calculate the gradient of each trainable parameter in the model (such as convolutional kernel weights, fully connected layer weights, bias terms, etc.). The adaptive moment estimation optimizer uses the first-moment estimate (mean) and second-moment estimate (uncentered variance) of each parameter's gradient to dynamically adjust the learning rate for each parameter. The update rule of the adaptive moment estimation optimizer combines the advantages of momentum and adaptive learning rates, effectively finding the minimum point of the composite loss function in the complex parameter space, thereby progressively updating the weight parameters of the initial multimodal deep learning model.

[0053] In practice, after obtaining the trained deep learning model, a model lightweighting step is performed. The parameter distribution of each branch in the trained deep learning model is analyzed to identify the top convolutional layers with the highest parameter percentage. Specifically, this analysis involves counting the number of trainable parameters in each convolutional layer, including the kernel weights and biases. The convolutional layers are then sorted from highest to lowest parameter count, and the fewest convolutional layers whose total parameters reach a specific percentage (e.g., 70%) are selected as the top convolutional layers with the highest parameter percentage. These layers are typically deep convolutional layers with a large number of channels and are the primary targets for model compression.

[0054] In practice, a channel pruning algorithm is used to evaluate the output channels of the first few convolutional layers, removing redundant channels whose contribution is below a pruning threshold. The algorithm calculates an importance score for each output channel of the convolutional layer being evaluated. This score can be based on the absolute mean of the channel's output feature map, the scaling factor of the batch normalization layer, or a contribution estimate based on a first-order Taylor expansion. A pruning threshold is set, and all channels with importance scores below this threshold are marked as redundant. In the model structure, these redundant channels and their corresponding input channel connections in subsequent layers are removed. Simultaneously, the weight matrices of relevant layers are adjusted, and rows and columns associated with the pruned channels are removed. The channel pruning algorithm reduces the number of channels and parameters in the model.

[0055] In some embodiments, the pruned network structure undergoes reconstruction of fully connected layers, replacing the original fully connected layers with grouped convolutional connections. Fully connected layers are typically found in classification decision heads, where the input and output neurons are fully connected. The reconstruction operation equivalently transforms the fully connected layer into a 1x1 convolutional layer. This standard 1x1 convolutional layer is then further replaced with grouped convolutional layers, dividing the input and output channels into multiple groups. Channels within a group are fully connected, while channels between groups are unconnected. The number of groups is a preset hyperparameter; grouped convolutional connections significantly reduce the number of parameters and computational cost of the fully connected layer. Optionally, a channel shuffling operation can be added after the grouped convolutional layers to facilitate information exchange between different groups.

[0056] In practice, the pruned and reconstructed network structure and weight files are exported to replace the original trained deep learning model, resulting in a lightweight inference model suitable for embedded devices. The export process includes saving the new, more compact model definition file and the corresponding weight parameter file. The weight parameters originate from the original trained deep learning model but have been adjusted through pruning and reconstruction. The resulting lightweight inference model maintains the main functionality of the original model while having fewer parameters, lower memory usage, and faster computation speed, thus enabling deployment on resource-constrained embedded devices to perform real-time multi-crop multimodal recognition tasks.

[0057] See Figure 5This is a cross-modal attention fusion layer structure diagram, which corely demonstrates how visible light, thermal infrared, and multispectral field crop data are intelligently fused and feature enhanced through an attention mechanism. It clearly shows the complete technical link of "input layer → computation layer → output layer". The first row is the original multimodal input, which is the three types of field observation data directly received by the model, representing real physical world observation values. The visible light feature map is presented as a yellow-green tone with a bright center and dark edges, representing the morphology and texture of the crop. The central area represents a healthy, vigorous crop canopy, and the dark edge area represents the background soil. The thermal infrared feature map is blue (low temperature) → green → yellow → red (high temperature), representing the temperature state of the crop. The high temperature area in the center corresponds to the active transpiration area of ​​the crop, reflecting the intensity of the crop's physiological metabolism. The multispectral feature map is yellow (low) → green → high (dark green), representing the growth vitality / biochemical components of the crop (such as the NDVI index). The high value in the center represents high vegetation cover and healthy growth. The second row represents the core calculation of the attention mechanism. By calculating similarity, weights are dynamically allocated to highlight key features. The query matrix Q, key matrix K, and value matrix V are presented in blue (negative) → white → red (positive). The query matrix Q is generated by projecting visible light features and is used to "search" for targets of interest in the feature space. The key matrix K is generated by projecting thermal infrared and multispectral features. It serves as the feature library to be searched. The value matrix V originates from K and stores the specific feature values ​​of thermal infrared and multispectral data, awaiting weighted retrieval. Attention weights are black (low weight) → white (high weight), representing the core result. Q and K are obtained by calculating similarity and then normalizing using Softmax. A highlighted center indicates that the model automatically identifies the core crop region, assigning it high attention; dark areas at the edges represent background soil, assigning it low weight. The third row is the fused feature output, applying the attention weights to the original features for precise enhancement. Physiological state enhancement features are presented in a yellow-green hue, with a bright center, representing visible light × attention weight. This preserves the crop's morphological texture while enhancing the physiological activity of high-weight regions, making the crop features more prominent. The joint feature tensor output is presented in a yellow-green hue with a bright center, and is the final output. Further fusing the enhanced physiological features with the original visible light features yields more discriminative joint features, which can be directly used for subsequent crop identification or classification tasks.

[0058] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multi-crop multimodal recognition method based on deep learning, characterized in that, include: Collect raw field crop datasets and label each sample in the raw field crop dataset with crop category labels and growth stage labels; The original field crop dataset is divided into a training subset and a validation subset, and the training subset is used to construct an initial multimodal deep learning model containing visible light convolution branches, thermal infrared convolution branches, and multispectral convolution branches. The training subset is input into the visible light convolution branch, the thermal infrared convolution branch, and the multispectral convolution branch respectively for feature extraction to obtain the visible light deep feature map, the thermal infrared deep feature map, and the spectral deep feature map. The visible light deep feature map, thermal infrared deep feature map, and spectral deep feature map are fed into the cross-modal attention fusion layer for weight recalibration to generate a joint feature tensor that integrates texture information and physiological state. The joint feature tensor is input into the temporal modeling unit to perform dynamic modeling of the growth cycle, and outputs a spatiotemporal correlation feature vector with a time dimension. The spatiotemporal correlation feature vector is fed into the classification decision head to determine the crop category and growth stage, generating a preliminary prediction result matrix. The preliminary prediction result matrix is ​​compared with the crop category label and growth stage label to calculate the backpropagation gradient and update the parameters of the initial multimodal deep learning model until the model converges, thus obtaining the trained deep learning model.

2. The deep learning-based multi-crop multimodal recognition method according to claim 1, characterized in that, The original field crop dataset is divided into a training subset and a validation subset, and an initial multimodal deep learning model is constructed using the training subset, including: The original field crop dataset includes visible light images, thermal infrared images, and multispectral data. A stratified sampling strategy is adopted. According to the sample proportions of different crop categories and different growth stages in the original field crop dataset, samples are randomly drawn from each category-stage combination according to a preset proportion in order to maintain the original data distribution and construct the training subset and validation subset. For the visible light convolutional branch, an encoder structure based on stacked depthwise separable convolutional blocks is constructed, wherein each depthwise separable convolutional block contains a depthwise convolutional layer and a pointwise convolutional layer, used to extract spatial texture features from the visible light image; For the thermal infrared convolution branch, an encoder structure including a hollow spatial pyramid pooling module is constructed. The hollow spatial pyramid pooling module is configured with multiple convolutional layers with different void ratios in parallel to capture multi-scale thermal radiation features from thermal infrared images. For the multispectral convolution branch, a band feature encoder containing a one-dimensional convolutional layer is constructed. The one-dimensional convolutional layer performs convolution operations in the spectral dimension to extract spectral reflectance features from the multispectral band data. The outputs of the visible light convolutional branch, the thermal infrared convolutional branch, and the multispectral convolutional branch are connected through a shared cross-modal feature alignment layer. The cross-modal feature alignment layer maps the feature maps output by the three branches to a unified feature dimension through a learnable linear projection matrix, thereby forming the initial multimodal deep learning model.

3. The deep learning-based multimodal recognition method according to claim 1, characterized in that, After dividing the original field crop dataset into training and validation subsets, the data cleaning step is also included: Iterate through each sample in the training subset and extract the sharpness score of the visible light image and the signal-to-noise ratio of the thermal infrared image of the sample. Set a sharpness score threshold and a signal-to-noise ratio (SNR) index threshold, and mark samples whose sharpness score is lower than the sharpness score threshold or whose SNR index is lower than the SNR index threshold as low-quality samples; All low-quality samples in the training subset are removed, and the missing positions after removal are filled with neighboring samples using an interpolation algorithm to ensure the data integrity of the training subset. The cleaned training subset is then input back into the initial multimodal deep learning model for subsequent training.

4. The deep learning-based multimodal recognition method according to claim 1, characterized in that, The training subset is input into the visible light convolution branch, the thermal infrared convolution branch, and the multispectral convolution branch respectively for feature extraction, including: For the visible light convolutional branch, a depth-separable convolutional layer is used to replace the standard convolutional layer, and spatial feature dimensionality reduction is performed on the visible light images in the training subset to obtain low-resolution visible light spatial features. For the aforementioned thermal infrared convolution branch, a hollow spatial pyramid pooling structure is introduced to extract the physiological features of the multi-scale receptive field from the thermal infrared images in the training subset, thereby obtaining the thermal features characterizing the crop transpiration state. For the multispectral convolution branch, the differences between the multispectral bands in the training subset are calculated to generate a vegetation index feature map, and the vegetation index feature map is convolutionally encoded to obtain biochemical component features. The visible light spatial features, thermal features, and biochemical component features are normalized and adjusted to a uniform tensor size before being output.

5. The multi-crop multimodal recognition method based on deep learning according to claim 1, characterized in that, The visible light deep feature map, thermal infrared deep feature map, and spectral deep feature map are fed into a cross-modal attention fusion layer for weight recalibration, generating a joint feature tensor that integrates texture information and physiological state, including: In the cross-modal attention fusion layer, a query matrix, a key matrix, and a value matrix are constructed, wherein the query matrix is ​​generated by the visible light deep feature map, and the key matrix is ​​formed by splicing the thermal infrared deep feature map and the spectral deep feature map; Calculate the relevance score between the query matrix and the key matrix, and generate an attention weight distribution map based on the relevance score; The value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation of enhanced physiological state. The enhanced feature representation of the physiological state is concatenated with the visible light deep feature map through channels, and the channel dimension is compressed through convolution operation to generate the joint feature tensor.

6. The multi-crop multimodal recognition method based on deep learning according to claim 5, characterized in that, The value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation of enhanced physiological state, including: Extract the multimodal attention weight vector corresponding to the thermal infrared deep feature map and the spectral deep feature map from the attention weight distribution map; The multimodal attention weight vector is multiplied element-wise with the eigenvalues ​​at the corresponding positions in the value matrix to obtain the weighted thermal infrared feature blocks and spectral feature blocks. The weighted thermal infrared feature blocks and spectral feature blocks are spliced ​​along the channel dimension to form a temporary enhanced feature set; A cross-channel weighted average calculation is performed on the features in the enhanced feature set to generate a physiological state enhanced feature representation that reflects the intensity of crop transpiration and nutrient absorption status.

7. The multi-crop multimodal recognition method based on deep learning according to claim 1, characterized in that, The joint feature tensor is input into the temporal modeling unit for dynamic modeling of the growth cycle, and outputs a spatiotemporal correlation feature vector with a time dimension, including: The joint feature tensor is reshaped into a time-series sample sequence, where each element of the time-series sample sequence corresponds to the crop feature state at a specific acquisition time. The time-series sample sequence consists of multiple data points from the same field location at different collection times. Each data point contains visible light, thermal infrared, and multispectral information for that location at the corresponding time. The time-series sample sequence is input into a bidirectional gated loop unit for forward and backward information transmission to capture the continuous change pattern of crop growth. At each time step, the hidden state of the bidirectional gated loop unit is extracted and residually connected with the original joint feature tensor corresponding to the time step. Global max pooling is performed on the features processed at all time steps to aggregate them and obtain the spatiotemporal correlation feature vector with time dimension.

8. The deep learning-based multimodal recognition method according to claim 1, characterized in that, The preliminary prediction matrix is ​​compared with the crop category labels and growth stage labels to calculate the backpropagation gradient and update the parameters of the initial multimodal deep learning model, including: The crop category label is converted into a real category vector in one-hot encoded form, and the growth stage label is converted into a serialized growth stage vector. The preliminary prediction result matrix is ​​split into crop category prediction vectors and growth stage prediction vectors; Calculate the cross-entropy loss between the crop category prediction vector and the true category vector, and the mean squared error loss between the growth stage prediction vector and the growth stage vector, respectively. The cross-entropy loss value and the mean squared error loss value are weighted and summed according to a preset scaling factor to obtain the composite loss function value. The partial derivatives of all weights in the initial multimodal deep learning model are calculated based on the composite loss function value, and the weight parameters are updated using an adaptive moment estimator optimizer.

9. The deep learning-based multimodal recognition method according to claim 1, characterized in that, After obtaining the trained deep learning model, a model lightweighting step is also included: Analyze the parameter distribution of each branch in the trained deep learning model and identify the top few convolutional layers with the highest parameter proportions. The output channels of the first few convolutional layers are evaluated using a channel pruning algorithm, and redundant channels with a contribution lower than the pruning threshold are removed. The pruned network structure is reconstructed by using fully connected layers, replacing the original fully connected method with a grouped convolutional connection method. Export the pruned and reconstructed network structure and weight files, replace the original trained deep learning model, and obtain a lightweight inference model suitable for embedded devices.

10. The deep learning-based multi-crop multimodal recognition method according to claim 5, characterized in that, The value matrix is ​​weighted and summed using the attention weight distribution map to generate a feature representation of enhanced physiological state, including: Extract the first modal attention weight vector corresponding to the thermal infrared deep feature map and the second modal attention weight vector corresponding to the spectral deep feature map from the attention weight distribution map; The first modality attention weight vector is multiplied element-wise with the feature values ​​from the thermal infrared deep feature map in the value matrix to obtain the weighted thermal infrared feature block; The second modality attention weight vector is multiplied element-wise with the feature values ​​from the deep spectral feature map in the value matrix to obtain the weighted spectral feature blocks; The weighted thermal infrared feature blocks and the weighted spectral feature blocks are spliced ​​along the channel dimension to form an enhanced feature set; A global average pooling operation is performed on the enhanced feature set along the channel dimension to generate a single-channel global feature representation that reflects the physiological state of the crop. The single-channel global feature representation is extended along the spatial dimension and fused with the weighted thermal infrared feature blocks and the weighted spectral feature blocks by dot product to generate the final physiological state enhanced feature representation.

Images