A high-temperature-resistant and high-salt-resistant laccase from bacillus subtilis

Abstract

This invention relates to the field of predicting minor water loss in highland barley plants, specifically to a method for predicting minor water loss in highland barley plants based on DETR multimodal fusion. The technical solution includes: sampling minor water loss in highland barley plants and acquiring multimodal data to obtain an original dataset containing flag leaf hyperspectral data, whole-plant RGB data, and true water content labels; preprocessing the flag leaf hyperspectral data and RGB image data in the original dataset; extracting one-dimensional spectral deep features related to water content within the flag leaf and two-dimensional visual deep features related to water stress in the whole-plant canopy; fusing multimodal features based on a DETR encoder to output multimodal global features; constructing an improved DETR regression model; training the improved DETR regression model using the original dataset; and using the trained model to predict minor water loss in highland barley plants. This invention is applicable to predicting minor water loss in highland barley plants.

Description

Technical Field

[0001] This invention relates to the field of low-loss prediction of barley plant water content, specifically to a method for low-loss prediction of barley plant water content based on DETR (DetectionTransformer, an end-to-end target detection model based on Transformer) multimodal fusion. Background Technology

[0002] Highland barley is a core food crop in the Ganzi Tibetan Autonomous Prefecture, and the plant water content is a key indicator for judging drought stress and guiding precision irrigation.

[0003] The main methods for detecting the moisture content of highland barley are as follows:

[0004] Drying and weighing method: highly destructive, time-consuming, and cannot provide real-time guidance for irrigation.

[0005] Indirect method of soil moisture: can only measure soil moisture and cannot reflect the true water shortage status of plants.

[0006] Single-modal vision method: using only RGB or only spectrum, with insufficient accuracy and easily affected by the environment.

[0007] Existing models are mostly simple splicing and fusion, without using the DETR global attention structure, resulting in inaccurate prediction of barley moisture content.

[0008] The existing technology has the following technical defects and shortcomings:

[0009] Single-modal data lacks sufficient feature dimensions, limiting prediction accuracy. Relying solely on hyperspectral data is susceptible to leaf condition and noise interference; relying solely on RGB images cannot reflect the physiological information of water content within the leaves, resulting in large prediction errors and poor robustness.

[0010] Multimodal data is simply stitched together, resulting in insufficient fusion. Existing technologies mostly employ direct feature concatenation or shallow fusion, failing to achieve deep interaction and complementarity between spectral and visual features, thus failing to uncover the correlation information between hyperspectral and RGB data, and resulting in weak model generalization ability.

[0011] Traditional models lack the ability to model global features. Methods such as CNN (Convolutional Neural Network), PLSR (Partial Least Squares Regression), and random forest can only extract local features and cannot establish long-range dependencies between plant canopy, leaf features, and water content, resulting in insufficient prediction stability in complex field environments.

[0012] The system lacks a dedicated sampling procedure for slightly damaged barley flag leaves, which does not match the actual planting and rapid field testing needs of barley, resulting in poor applicability.

[0013] The model structure is mismatched with the task, making deployment difficult. Existing models are mostly designed for general crops or target detection, and have not been modified to a water content regression prediction structure. The models are large in size and slow inference, making them unsuitable for real-time operation of field edge devices. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for predicting the water content of barley plants with minimal loss based on DETR multimodal fusion, thereby improving the accuracy of the prediction of the water content of barley plants with minimal loss.

[0015] This invention achieves the above objectives by adopting the following technical solution: This invention provides a method for predicting the water content of highland barley plants with minimal loss based on DETR multimodal fusion, comprising:

[0016] Includes the following steps:

[0017] S1. Minimally damaged sampling and multimodal data acquisition of highland barley plants;

[0018] Select barley plants to be tested and assign them unique numbers to ensure that subsequent data correspond one-to-one. For each numbered barley plant, remove the top healthy flag leaf with minimal damage to complete the minimally damaged sampling.

[0019] The flag leaf was laid flat on the stage, and hyperspectral data of the flag leaf was collected under a standard light source to generate a continuous one-dimensional sequence of reflectance data from the visible light to the near-infrared band.

[0020] The whole plant of the same barley plant was photographed to obtain two-dimensional RGB image data including plant shape, color and growth status;

[0021] The flag leaf hyperspectral data collected according to the barley plant number was paired with RGB image data to obtain the original dataset containing flag leaf hyperspectral data, whole plant RGB data, and true water content labels.

[0022] S2. Preprocess the flag leaf hyperspectral data and RGB image data in the original dataset;

[0023] S3. Perform dual-modal feature extraction on the preprocessed data to extract one-dimensional spectral deep features related to water content inside the flag leaf and two-dimensional visual deep features related to water stress in the whole canopy.

[0024] S4. Multimodal feature fusion based on DETR encoder;

[0025] One-dimensional spectral deep features and two-dimensional visual deep features are simultaneously input into a multimodal feature fusion module based on a modified DETR encoder. Through the cross-attention mechanism of the multimodal feature fusion module, deep interaction and global modeling of one-dimensional spectral deep features and two-dimensional visual deep features are realized. One-dimensional spectral deep features supplement the details of water content inside the leaves, and two-dimensional visual deep features supplement the global information of the overall plant growth. After fusion, multimodal global features are output.

[0026] S5. Construct an improved DETR regression model;

[0027] The standard DETR target detection model is modified by removing the original target detection head and replacing it with a regression prediction structure, and introducing a learnable regression query vector to adapt to the water content regression task.

[0028] The Transformer encoder and decoder structure is retained, and the decoder completes feature modeling through self-attention, cross-attention, and fully connected layers;

[0029] A regression prediction head, i.e. a fully connected network, is connected after the decoder to map the fused multimodal global features into continuous numerical output, realizing end-to-end prediction from features to water content.

[0030] S6. Train the improved DETR regression model using the original dataset, and use the trained model to predict the slight loss of water content in highland barley plants.

[0031] Furthermore, step S2 specifically includes:

[0032] Flag leaf hyperspectral data preprocessing: Black and white correction, noise removal, outlier removal and reflectance normalization are performed on the original flag leaf hyperspectral data. The continuous projection algorithm is used to screen the feature bands that are highly correlated with the water content of barley, remove redundant information, and output a fixed-dimensional hyperspectral feature vector.

[0033] RGB image data preprocessing: Perform size normalization, dehazing enhancement, background removal and pixel value normalization on the original RGB image data, unify the image input format, and eliminate lighting and background interference;

[0034] Complete the time synchronization, spatial alignment and sample pairing of flag leaf hyperspectral data and RGB image data to ensure that the hyperspectral features and RGB features correspond to the same plant.

[0035] Furthermore, step S3 specifically includes:

[0036] Flag leaf hyperspectral feature extraction: The preprocessed hyperspectral feature vector is encoded by an encoder of a fully connected network to extract one-dimensional deep spectral features related to water content inside the flag leaf.

[0037] RGB visual feature extraction: The preprocessed RGB image is encoded using CNN and Transformer encoders to extract two-dimensional visual deep features of the whole canopy related to water stress.

[0038] Furthermore, step S6 specifically includes:

[0039] The original dataset is divided into training set, validation set and test set according to a set ratio, and then input into the model for training.

[0040] We use a weighted combination of two loss functions, mean squared error and mean absolute error, as the model optimization objective to balance prediction bias and accuracy.

[0041] The AdamW optimizer is used to update the model parameters, combined with an early stopping strategy to prevent overfitting. R², RMSE and CORR are used as the core evaluation indicators to iteratively optimize the model parameters until the model reaches the optimal performance on the validation set.

[0042] After training is completed, the optimal model weights are saved, and the water content of barley plants is predicted with minimal loss.

[0043] Furthermore, the method also includes:

[0044] S7. Model Validation and Performance Evaluation;

[0045] The trained model was independently validated using a test set. Hyperspectral and RGB dual-modal data that were not used in the training were input to obtain water content prediction results.

[0046] The model's R², RMSE, and MAE indices were calculated by comparing the actual moisture content calibrated by the drying method to verify the model's prediction accuracy, robustness, and generalization ability.

[0047] Specialized tests were conducted on barley samples at different growth stages and under different levels of drought stress to verify the model's adaptability in complex highland field conditions and ensure that the model meets actual testing needs.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention simultaneously inputs flag leaf hyperspectral data (reflecting the internal water physiology of the leaf) and whole-plant RGB image (reflecting the plant's appearance and growth). It achieves deep fusion of dual-modal features through a DETR encoder and cross-attention mechanism, replacing the traditional single-modal input and simple splicing fusion. This significantly improves the model's prediction accuracy, effectively eliminates the interference of light, background, and growth stage changes on the detection results, and significantly enhances the model's robustness.

[0050] This invention is the first to transfer the DETR architecture from target detection tasks to barley moisture content regression prediction tasks. Through the Transformer global attention mechanism, it establishes a long-range dependency between leaf spectral features and whole-plant visual features, realizes global modeling and deep interaction of dual-modal features, fully explores the complementary information of the two types of data, and significantly improves the model's generalization ability. It can be adapted to barley samples of different growth stages and different drought stress levels in Ganzi Tibetan Autonomous Prefecture.

[0051] This invention employs a minimally invasive sampling method involving the removal of flag leaves, balancing detection accuracy with the impact on plant growth. Only a small number of functional leaves are collected, resulting in virtually no impact on the subsequent growth of the barley, far superior to the completely destructive detection methods of traditional drying. The entire process of sampling, collection, and inference is fast, and the detection efficiency is superior to traditional drying methods.

[0052] This invention constructs a dedicated end-to-end process of "flag leaf micro-destructive sampling → hyperspectral acquisition → whole plant RGB imaging → dual-modal pairing → model prediction", which perfectly matches the actual testing needs in highland barley fields.

[0053] This invention is the first to combine micro-destructive sampling with the DETR multimodal fusion model to form a dedicated detection solution for highland barley, solving the problems of generalized design and incompatibility with highland barley planting scenarios in existing technologies.

[0054] The invention features a standardized process and simple operation, allowing field operators to complete the testing with minimal training, significantly reducing the barrier to technology implementation. Attached Figure Description

[0055] Figure 1 This is a flowchart of a method for predicting the water content of barley plants with minimal loss based on DETR multimodal fusion, provided by an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0057] This invention provides a method for predicting the water content of barley plants with minimal loss based on DETR multimodal fusion, such as... Figure 1 As shown, it specifically includes:

[0058] S1. Minimally damaged sampling and multimodal data acquisition of highland barley plants;

[0059] In the barley fields of Ganzi Tibetan Autonomous Prefecture, plants to be tested were selected and uniquely numbered to ensure that subsequent data corresponded one by one. The top healthy flag leaves of the numbered barley plants were slightly damaged and removed (only a small number of functional leaves were taken, which would not affect the subsequent growth of the plant) to complete the minimally damaged sampling.

[0060] The flag leaf was laid flat on the stage, and hyperspectral data of the flag leaf was collected under a standard light source to generate a continuous one-dimensional sequence of reflectance data from the visible light to the near-infrared band.

[0061] The same barley plant under test was photographed by an RGB image acquisition terminal to obtain two-dimensional RGB image data containing plant shape, color and growth status.

[0062] The flag leaf hyperspectral data collected according to the barley plant number was paired with the RGB image data to obtain the original dataset containing flag leaf hyperspectral data, whole plant RGB data, and true water content labels.

[0063] S2. Preprocess the flag leaf hyperspectral data and RGB image data in the original dataset;

[0064] Hyperspectral data preprocessing: Black and white correction, noise removal, outlier removal and reflectance normalization are performed on the original flag leaf hyperspectral data; the continuous projection algorithm is used to screen feature bands that are highly correlated with the water content of barley, remove redundant information, and output a fixed-dimensional hyperspectral feature vector;

[0065] RGB image data preprocessing: Perform size normalization, dehazing enhancement, background removal and pixel value normalization on RGB images to unify the image input format and eliminate lighting and background interference;

[0066] The time synchronization, spatial alignment and sample pairing of the dual-modal data were completed to ensure that the hyperspectral features and RGB features strictly corresponded to the same plant, thus preparing for subsequent feature extraction.

[0067] S3. Perform dual-modal feature extraction on the preprocessed data to extract one-dimensional spectral deep features related to water content inside the flag leaf and two-dimensional visual deep features related to water stress in the whole canopy.

[0068] Flag leaf hyperspectral feature extraction: The preprocessed hyperspectral feature vector is encoded by an encoder of a one-dimensional fully connected network to extract one-dimensional deep spectral features related to water content inside the flag leaf.

[0069] In one embodiment of the present invention, the one-dimensional fully connected network encoder is specifically as follows:

[0070] The input is the preprocessed hyperspectral feature vector. The original hyperspectral data contains approximately 200 bands, which are reduced to 21 key bands after selection using the Continuous Projection (SPA) algorithm. The input tensor shape is (B, n_bands), where B is the batch size and n_bands defaults to 21. During the preprocessing stage, each band is also normalized using the Z-score method to achieve a mean of 0 and a standard deviation of 1.

[0071] Detailed Explanation of One-Dimensional Fully Connected Network Structure:

[0072] A one-dimensional fully connected network encoder consists of the following four parts connected in series:

[0073] The Band Projection layer projects the scalar value of each band onto a high-dimensional feature space. Its specific structure is nn.Linear(1, d_model), where the input dimension is 1 (the scalar value of a single band), and the output dimension is d_model = 256. This operation is equivalent to performing an independent embedding transformation on each band, mapping the one-dimensional scalar to a 256-dimensional feature vector. After this layer, the data shape is transformed from (B, 21) to (B, 21, 256).

[0074] Learned Positional Encoding is introduced in this encoder to preserve the order information of each band in the spectral sequence. This encoding is a trainable parameter nn.Parameter with a shape of (1, 21, 256), meaning that each band position corresponds to a 256-dimensional learnable vector.

[0075] The initialization uses a truncated normal distribution (trunc_normal_, standard deviation 0.02). Unlike sinusoidal position encoding, this encoder uses learnable position embeddings, enabling the model to adaptively learn the relative positional relationships between bands from the data.

[0076] The feedforward network block (FFNBlock) consists of two stacked feedforward network blocks with the same structure. Each FFNBlock uses a pre-normalized architecture, and its internal structure is as follows:

[0077] Layer normalization (LayerNorm) with dimension d_model = 256;

[0078] The first fully connected layer maps the dimension from 256 to 1024;

[0079] Dropout layer, drop rate 0.1;

[0080] The second fully connected layer maps the dimension back from 1024 to 256;

[0081] Dropout layer, drop rate 0.1;

[0082] The activation function is ReLU, located after the first fully connected layer. Each FFNBlock also contains residual connections, meaning the input features are directly added to the FFN output before entering the next layer.

[0083] The residual connection and normalization strategy ensures that the overall network follows the pre-normalization design of the standard Transformer. Residual connections are set in each sub-layer (after the band projection layer and inside the FFNBlock) to ensure smooth gradient propagation and avoid degradation in deep networks.

[0084] The encoder outputs a final dimension of (B, 21, 256), meaning each band position corresponds to a 256-dimensional deep feature vector. This output can be directly used for subsequent multimodal fusion, or it can be first transformed into a compact feature vector of (B, 256) through global average pooling (averaging along the band dimensions).

[0085] In a lightweight version (optional), in another embodiment, the encoder can employ a one-dimensional convolutional neural network architecture as a lightweight alternative. Its structure is a progressive dimensionality increase from 1 to 32 to 64 to 128 to 256, with each convolutional layer followed by batch normalization (BatchNorm), GELU activation function, and max pooling with a stride of 2. Furthermore, the input features are mapped to 256 dimensions using 1×1 convolutions and then added to the output to form residual connections. Finally, adaptive average pooling compresses the sequence length to 1, outputting global features of shape (B, 256).

[0086] The encoder was trained using the following hyperparameter settings:

[0087] Optimizer: AdamW, initial learning rate 1e-4, weight decay coefficient 0.05;

[0088] Learning rate decay strategy: multiply by 0.1 every 80 training cycles;

[0089] Dropout ratio: 0.1, applied within the feedforward network block;

[0090] Normalization method: Layer normalization was used, batch normalization was not used;

[0091] Regularization: Relies on Dropout and weight decay, without using any other explicit regularization methods.

[0092] RGB visual feature extraction: The preprocessed RGB image is encoded using CNN and Transformer encoders to extract two-dimensional visual deep features of the whole canopy related to water stress.

[0093] In one embodiment of the present invention, the CNN portion is specifically as follows:

[0094] Default: ResNet50 (ImageNet pre-trained weights);

[0095] Optional: ResNet18 / ResNet34 / ResNet101 (configured via the --backbone parameter);

[0096] ResNet50 structure (number of output channels at each stage)

[0097] Input image: (B, 3, H, W) RGB 3 channels;

[0098] conv1: Convolution 7×7, stride 2, 64 channels → BatchNorm + ReLU + MaxPool(3×3, stride 2);

[0099] layer 1: 3 residual blocks, output 64 channels (downsample: 256→256).

[0100] layer 2: 4 residual blocks, output 512 channels (downsample: 256→512).

[0101] layer3: 6 residual blocks, output 1024 channels (downsample: 512→1024).

[0102] layer4: 3 residual blocks, outputting 2048 channels (downsample: 1024→2048).

[0103] Final output: (B, 2048, H / 32, W / 32).

[0104] Normalization method: FrozenBatchNorm2d: Freezes the affine parameters (gamma and beta) of BatchNorm and only updates the statistics (mean / variance). Used only during the pre-training phase.

[0105] Training strategy, default: layers 1-3 are frozen (when lr_backbone=1e-5), only layer 4 can be trained;

[0106] If lr_backbone=0, the entire backbone network is frozen;

[0107] In this embodiment: backbone_lr > 0, but lr_backbone = 1e-5 (very low learning rate);

[0108] Input preprocessing:

[0109] Image size: Minimum side scaled to min_size=448;

[0110] ImageNet normalization: mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225];

[0111] Data augmentation: During training, each RGB image undergoes random transformations such as horizontal flipping, color jittering, and random erasure to increase data diversity and prevent overfitting.

[0112] Location coding:

[0113] Default: Sine position encoding (PositionEmbeddingSine);

[0114] Optional: Learned position encoding (PositionEmbeddingLearned via --pretrained pos_embed_type);

[0115] This embodiment uses Sine Positional Encoding to encode the spatial location of the feature map. The specific parameters are as follows:

[0116] 1. Feature Dimension

[0117] Hidden layer dimension hidden_dim = 256;

[0118] Since the sine and cosine functions are used in pairs, 128 pairs of sin / cos codes are actually generated.

[0119] Each pair of codes corresponds to the sine and cosine values, and when concatenated, they form a complete 256-dimensional positional coding vector.

[0120] 2. Temperature parameters

[0121] Temperature value = 10000;

[0122] Function: To control the wavelength distribution of sine waves of different frequencies; the higher the temperature value, the higher the proportion of low-frequency components.

[0123] Calculation formula: Frequency freq = 1 / (10000^(2i / d)), where i is the dimension index;

[0124] 3. Normalization settings

[0125] Normalization switch normalize = True;

[0126] Function: Normalize all position indices (0, 1, 2, …, L-1) to the interval [0, 1];

[0127] Multiply by 2π to make the position value range [0, 0.5π] (actually falling in the interval [0, 1.57]);

[0128] Normalization formula: pos_normalized = pos / max_len × 2π;

[0129] 4. Output tensor shape

[0130] Full dimensions: (B, 256, H / 32, W / 32);

[0131] Meaning of each dimension:

[0132] B: Batch Size;

[0133] 256: Number of location encoding channels (equal to the hidden layer dimension);

[0134] H / 32: Feature map height (original image height downsampled by 32 times);

[0135] W / 32: Feature map width (downsampled 32 times from the original image width);

[0136] 5. Encoding Calculation Method

[0137] Position codes are generated using sine and cosine functions of different frequencies;

[0138] For each position coordinate (x, y), calculate the encoding in the row direction and column direction respectively;

[0139] 128 pairs of dimensions correspond to 128 different wavelengths, with high-frequency components corresponding to subtle positional changes and low-frequency components corresponding to global positional information;

[0140] The row code and column code are added or concatenated to form a complete 2D position code.

[0141] Multi-scale location coding:

[0142] BackboneBase supports outputting intermediate layer features:

[0143] layer2_out: (B, 512, H / 8, W / 8);

[0144] layer3_out: (B, 1024, H / 16, W / 16);

[0145] layer4_out: (B, 2048, H / 32, W / 32).

[0146] The Transformer encoder parameters are as follows:

[0147] d_model (hidden layer dimension): 256, the dimension of the feature vector.

[0148] nhead (number of attention heads): 8, the number of parallel heads in a multi-head self-attention mechanism.

[0149] num_encoder_layers (encoder layers): 4, the number of layers stacked in the Transformer encoder.

[0150] dim_feedforward (feedforward network dimension): 1024, the feature dimension of the intermediate layer of FFN.

[0151] dropout: 0.1, applied to the attention weights and the dropout ratio after the linear layer.

[0152] activation function: ReLU, the activation function used in feedforward networks.

[0153] normalize_before (normalization position): False, where False indicates post-normalization (original Transformer style), and True indicates pre-normalization (pre-Norm).

[0154] The Transformer encoder layer structure is as follows:

[0155] Each Transformer encoder layer follows this order: self-attention → addition and normalization → feedforward network → addition and normalization.

[0156] 1. Bullish Self-Attention

[0157] The input shape is (seq_len, B, 256), and the output shape is (seq_len, B, 256).

[0158] The query, key, and value all come from the same input sequence, but are transformed linearly using different projection matrices.

[0159] The dropout ratio is 0.1, applied to the attention weight matrix.

[0160] 2. Layer normalization and residual connectivity

[0161] After the self-attention sub-layer, the original input and the attention output are added together, and then layer normalization is performed.

[0162] 3. Feedforward network

[0163] Structure: Linear(256, 1024) → ReLU → Dropout(0.1) → Linear(1024, 256) → Dropout(0.1).

[0164] The entire feedforward network sublayer also contains residual connections, meaning the input is directly added to the FFN output.

[0165] 4. Second layer normalization and residual connectivity

[0166] After the feedforward network sub-layer, residual connections and layer normalization are performed again to obtain the final output of the current encoder layer.

[0167] Multi-head attention implementation details:

[0168] Multi-head attention is implemented using PyTorch's built-in nn.MultiheadAttention module.

[0169] batch_first = False, meaning the input tensor format is (seq_len, batch, d_model).

[0170] Attention dropout = 0.1 is the dropout rate applied to attention weights.

[0171] How CNN and Transformer are integrated:

[0172] This embodiment describes how the output feature map of a CNN is transformed into the sequence input required by the Transformer encoder.

[0173] Step 1: CNN Output

[0174] The layer 4 output shape of ResNet50 is (B, 2048, H / 32, W / 32).

[0175] Assuming the input image size is 448×448 pixels, the output shape is (B, 2048, 14, 14).

[0176] Step 2: Channel Projection

[0177] Use a 1×1 convolutional layer nn.Conv2d(2048, hidden_dim=256, kernel_size=1) to compress the number of channels from 2048 to 256.

[0178] This operation does not change the spatial dimensions of the feature map, and the output shape is (B, 256, 14, 14).

[0179] Step 3: Add location code

[0180] The shape of the sinusoidal position code is (1, 256, 14, 14).

[0181] The positional encoding is added element by element to the feature map, and the output shape remains (B, 256, 14, 14).

[0182] Step 4: Flatten into a sequence

[0183] Use flatten(2) to convert (B, 256, 14, 14) to (B, 256, 196).

[0184] Then use permute to rearrange the dimensions to (196, B, 256).

[0185] The sequence length seq_len = H / 32 × W / 32 = 14 × 14 = 196, and each spatial location corresponds to a 256-dimensional token.

[0186] Step 5: Input Transformer encoder

[0187] The sequence shape is (196, B, 256), which is (seq_len, batch, d_model).

[0188] Feature modeling is performed using four layers of self-attention encoders.

[0189] The encoder output shape remains (196, B, 256).

[0190] Step 6: Global Pooling

[0191] To obtain compact image-level features for subsequent multimodal fusion, the sequence dimension is averaged pooled: mean(dim=0), and the output shape is (B, 256).

[0192] In another alternative embodiment, a learnable [CLS] token can be used instead of average pooling. This token is input into the encoder along with the sequence, and the output at the corresponding position is taken as the global feature.

[0193] In this embodiment, the overall process for RGB visual feature extraction is as follows:

[0194] Step 1: Input RGB image

[0195] The tensor shape of the input RGB image is (B, 3, 448, 448), where B is the batch size, 3 is the number of color channels, and 448×448 is the image spatial resolution.

[0196] Step 2: Extracting features using the ResNet50 backbone network

[0197] An RGB image is input into the ResNet50 backbone network, and the output of its layer 4 is taken as the feature map. The shape of this feature map is (B, 2048, 14, 14), where 2048 is the number of channels and 14×14 is the spatial size of the feature map (i.e., the input image is downsampled by 32 times).

[0198] Step 3: Channel Projection

[0199] The number of channels in the feature map is compressed from 2048 to 256 using a 1×1 convolutional layer (input_proj). The output feature map shape is (B, 256, 14, 14).

[0200] Step 4: Overlay position encoding

[0201] The sinusoidal position code (with shape (B, 256, 14, 14)) is added element by element to the projected feature map to obtain a position-aware feature map carrying spatial position information. The shape remains unchanged and is still (B, 256, 14, 14).

[0202] Step 5: Flatten into a sequence

[0203] The location-aware feature map is flattened and rearranged in dimensions: First, it is flattened along the spatial dimension, transforming (B, 256, 14, 14) into (B, 256, 196) (where 196 = 14 × 14, which is the total number of spatial locations); then, it is transformed into (196, B, 256) through dimensional permutation, that is, the sequence length is 196, and each location corresponds to a 256-dimensional token.

[0204] Step 6: Transformer encoder modeling

[0205] The above sequence is input into a Transformer encoder. The encoder consists of four layers, each composed of a multi-head self-attention network, a feedforward network, and residual connections. The input sequence has the shape (196, B, 256), and the output shape remains unchanged at (196, B, 256).

[0206] Step 7: Global Pooling

[0207] The sequence output by the encoder is subjected to average pooling (mean calculated along the sequence dimension) to obtain a compact global feature vector with shape (B, 256). This vector serves as a deep feature of the RGB visual modality and participates in subsequent multimodal fusion.

[0208] The system outputs two independent deep features, corresponding to "internal leaf water physiological information" and "whole plant appearance and growth information", respectively, providing feature input for multimodal fusion.

[0209] S4. Multimodal feature fusion based on DETR encoder;

[0210] The one-dimensional spectral deep features and two-dimensional visual deep features are simultaneously input into a multimodal feature fusion module based on a modified DETR encoder. Through the cross-attention mechanism of the multimodal feature fusion module, deep interaction and global modeling of one-dimensional spectral deep features and two-dimensional visual deep features are realized. One-dimensional spectral deep features supplement the details of water content inside the leaves, and two-dimensional visual deep features supplement the global information of the overall plant growth. After fusion, multimodal global features are output.

[0211] Specifically, this embodiment implements three multimodal feature fusion strategies, which can be used individually or in combination:

[0212] Strategy A: EncoderCrossAttentionBlock: Performs fine-grained cross-modal interactions during the Transformer encoding phase.

[0213] Strategy B: MultimodalConcatAttentionFusion: Fusion is performed at the global feature level.

[0214] Strategy C: FusionGate: Perform final gating fusion before the regression prediction head.

[0215] The three strategies are explained in detail below.

[0216] Strategy A: EncoderCrossAttentionBlock (Cross attention based on a modified DETR encoder)

[0217] Based on the DETR encoder modification: The standard DETR encoder uses a unimodal self-attention mechanism, where all tokens perform attention calculations with each other. This invention modifies it into a bimodal cross-attention mechanism, specifically by designing EncoderCrossAttentionBlock, which uses two different types of token sequences as the query, key, and value, respectively.

[0218] Detailed Explanation of Cross-Attention Mechanism:

[0219] enter:

[0220] spec_seq: The output of the hyperspectral FFN encoder, with the shape (n_bands, B, 256) = (21, B, 256).

[0221] img_seq: The output of the RGB Transformer encoder, with a shape of (H×W, B, 256) = (196, B, 256).

[0222] Phase 1: Hyperspectral data as the query, RGB data as the key and value.

[0223] The query Q comes from a hyperspectral sequence: Q = spec_seq (21, B, 256);

[0224] Key K comes from the RGB sequence: K = img_seq (196, B, 256);

[0225] The value V comes from the RGB sequence: V = img_seq (196, B, 256);

[0226] Attention calculation: Attention(Q, K, V) = softmax(QK^T / sqrt(256)) × V;

[0227] Effect: For each hyperspectral band token (as a query), retrieve the most relevant part of the RGB visual features.

[0228] Output shape: (21, B, 256)

[0229] Phase 1.5: Hyperspectral Feedforward Network

[0230] Employing a pre-normalization structure: first, perform layer normalization on spec_seq;

[0231] Feedforward network structure: Linear(256, 2048) → ReLU → Linear(2048, 256);

[0232] Residual connection: spec_seq = spec_seq + FFN_output;

[0233] Phase 2: Updated hyperspectral data as the query, RGB as the key and value (second round)

[0234] The query Q is the output of stage 1, with shape (21, B, 256);

[0235] Key K is an RGB sequence with a shape of (196, B, 256);

[0236] The value V is an RGB sequence with shape (196, B, 256);

[0237] Effect: Further enhances the query and fusion of RGB information using hyperspectral imaging.

[0238] Stage 2.5: RGB Feedforward Network

[0239] The structure is the same as in stage 1.5.

[0240] Output:

[0241] The updated hyperspectral sequence spec_seq is (21, B, 256).

[0242] The updated RGB sequence img_seq is (196, B, 256). (Note: Although the RGB sequence was not directly modified during cross-attention, the hyperspectral sequence has been fully incorporating information from RGB after two-stage interaction.)

[0243] Key design notes:

[0244] Cross-attention is unidirectional: only the hyperspectral sequence queries the RGB sequence, and the RGB sequence itself is not directly modified.

[0245] The physical meaning of this design is to allow each band of the hyperspectral spectrum to "focus" on the most relevant region in the RGB visual space, thereby establishing a correspondence between spectral bands and visual regions.

[0246] The feedforward network layer ensures that each mode has nonlinear transformation capabilities.

[0247] The number of cross-attention blocks, num_cross_blocks, defaults to 1 and can be adjusted via configuration parameters.

[0248] In addition to obtaining global features through average pooling, this embodiment can also map the hyperspectral sequence spec_seq_updated to a vector spec_feat of (B, 256) through a self-attention layer (spec_attn) and a projection layer (spec_proj) for subsequent gated fusion.

[0249] Strategy B: MultimodalConcatAttentionFusion (Concatenation and Token Attention)

[0250] enter:

[0251] spec_global: The hyperspectral encoder output is obtained by global average pooling, with a shape of (B, 256).

[0252] img_global: The RGB encoder output is obtained by global average pooling, with a shape of (B, 256);

[0253] Fusion process:

[0254] Step 1: Global Average Pooling

[0255] The hyperspectral sequence spec_seq (21, B, 256) is averaged along the sequence dimension to obtain spec_global(B, 256);

[0256] The mean of the RGB sequence img_seq (196, B, 256) along the sequence dimension is used to obtain img_global (B,256).

[0257] Step 2: Assembling

[0258] Concatenate two global feature vectors along the feature dimension: cat = torch.cat([spec_global, img_global], dim=1);

[0259] The shape after splicing is (B, 512).

[0260] Step 3: Shared Blending Projection

[0261] Pre-normalization: Perform layer-level normalization on the spliced ​​features;

[0262] The first linear layer: Linear(512, 256), compresses the dimensions;

[0263] Activation function: GELU;

[0264] Dropout: 0.1;

[0265] The second linear layer: Linear(256, 256), projected again;

[0266] Output: fused features (B, 256);

[0267] This step fuses the information from the two modalities at a semi-semantic level.

[0268] Step 4: Memorize the token

[0269] Set num_memory_tokens = 4 learnable memory tokens (query tokens);

[0270] Shape: (4, B, 256), all tokens have the same initial value but their gradients are updated independently;

[0271] These memory tokens aggregate information from [spec_global, img_global, fused] through an attention mechanism.

[0272] Step 5: Cross-attention (memory tokens focus on three sources)

[0273] Query Q: Memory token, shape (4, B, 256);

[0274] Key K: Concatenate spec_global, img_global, and fused into (3, B, 256);

[0275] Value V: Same as bond K;

[0276] A multi-head attention mechanism is employed, where queries are derived from memory tokens, and keys and values ​​are derived from concatenated multimodal features. (Number of attention heads = 8)

[0277] Output: memory_output (4, B, 256);

[0278] This output serves as the decoder's memory input.

[0279] Output:

[0280] fused_memory: Shape (4, B, 256) contains 4 memory tokens that encode fusion information from hyperspectral and RGB.

[0281] Strategy C: FusionGate

[0282] Location: This module is located before the regression prediction head and is used for adaptive weighted fusion of visual and hyperspectral features. Its specific structure is: a linear layer with an input dimension of 512 (the dimension of the concatenated visual and hyperspectral features) and an output dimension of 256; followed by a Sigmoid activation function. The calculation process is as follows:

[0283] First, the visual feature `vis_feat` and the hyperspectral feature `spec_feat` are concatenated along the feature dimension to obtain a combined feature with dimension 512. Then, this combined feature is input into a linear layer and a sigmoid activation function to calculate the gate value:

[0284] gate = Sigmoid(Linear(Concat(vis_feat, spec_feat)))

[0285] Finally, the two features are weighted and fused using the gate value to obtain the fused feature fused_feat:

[0286] fused_feat = gate * vis_feat + (1 - gate) * spec_feat

[0287] Here, * indicates element-wise multiplication. The gating value ranges from 0 to 1, and is calculated independently for each sample, thus adaptively determining the contribution ratio of visual features and hyperspectral features in the final prediction.

[0288] This embodiment includes a gated fusion module preceding the regression prediction head to adaptively fuse RGB visual features and hyperspectral features. The specific structure of this module is as follows:

[0289] Let the RGB visual features be Hyperspectral characteristics are First, the two are concatenated along the feature dimension to obtain... Then, the concatenated features are input into a linear layer, mapping their dimension from 512 back to 256, and then passed through a Sigmoid activation function to obtain the gate value. :

[0290] ;

[0291] in, and For learnable parameters, This is the Sigmoid function.

[0292] Ultimately, gating fusion features Calculate using the following formula:

[0293] ;

[0294] in, This indicates element-wise multiplication.

[0295] In a specific implementation example, the above structure can be implemented using the following PyTorch code:

[0296] nn.Sequential(

[0297] nn.Linear(512, 256),

[0298] nn.Sigmoid(), )

[0300] Calculation process:

[0301] Gating value calculation: gate = sigmoid(MLP([vis_feat; spec_feat]));

[0302] Fusion feature: fused = gate × vis_feat + (1 - gate) × spec_feat;

[0303] Physical meaning:

[0304] The gating value determines the relative importance of RGB visual features and hyperspectral features in the final prediction.

[0305] The gating value can be different for each sample, and adaptive learning is driven by data.

[0306] The Sigmoid function outputs a value between 0 and 1, making the fusion a weighted average of the two features.

[0307] DETR decoder section:

[0308] Learnable query vectors:

[0309] The default value for num_queries is 16, which means there are 16 learnable query tokens with a shape of (16, B, 256).

[0310] These query vectors focus on the fused memory feature map through a cross-attention mechanism.

[0311] Decoder layer structure:

[0312] Decoder layers: 6.

[0313] Each layer of the structure includes:

[0314] Self-attention: Self-attention is calculated between query vectors.

[0315] Cross attention: The query vector focuses on the fused memory feature map.

[0316] Optionally, each layer can output auxiliary predictions (configured via --aux_loss).

[0317] Regression prediction head:

[0318] The regression prediction head is used to map the fused multimodal features to the final water content prediction value. This embodiment uses a multilayer perceptron structure as the prediction head, with the fused features output from the gated fusion module as its input. The output is the predicted flag leaf water content. (Unit: percentage).

[0319] The prediction head consists of three stacked fully connected layers, interspersed with GELU activation functions and Dropout layers, as shown in the following structure:

[0320] The first layer has an input dimension of 256 and an output dimension of 256, which is used to perform non-linear transformations while maintaining the feature dimensions.

[0321] The second layer has an input dimension of 256 and an output dimension of 64. It progressively compresses the feature dimensions to extract higher-level abstract features.

[0322] The third layer has an input dimension of 64 and an output dimension of 1, which maps the compressed features to a single water content prediction value.

[0323] Each fully connected layer is followed by the GELU activation function (Gaussian Error Linear Unit). ,in The cumulative distribution function of the standard normal distribution is used, and a Dropout layer (dropout rate 0.1) is used to introduce nonlinearity and prevent overfitting.

[0324] The above structure can be represented by the following formula:

[0325] ;

[0326] ;

[0327] ;

[0328] in, , , For learnable weight matrix, , , This is a bias term.

[0329] As a concrete code implementation example, the above structure can be constructed using the following PyTorch code:

[0330] nn.Sequential(

[0331] nn.Linear(256, 256),

[0332] nn.GELU(),

[0333] nn.Dropout(0.1),

[0334] nn.Linear(256, 64),

[0335] nn.GELU(),

[0336] nn.Dropout(0.1),

[0337] nn.Linear(64, 1) )

[0339] It should be noted that the scope of protection of this invention is not limited to the specific code implementation described above, and any equivalent transformation based on the same structural principle falls within the scope of protection of this invention.

[0340] S5. Construct an improved DETR regression model;

[0341] The standard DETR target detection model is modified by removing the original target detection head and replacing it with a regression prediction-specific structure, and introducing a learnable regression query vector to adapt to the water content regression task.

[0342] The Transformer encoder and decoder structure is retained, and the decoder completes feature modeling through self-attention, cross-attention, and fully connected layers;

[0343] A regression prediction head, i.e., a fully connected network, is connected after the decoder to map the fused multimodal global features into continuous numerical outputs, thereby achieving end-to-end prediction from features to water content.

[0344] Based on the multimodal fusion features output in step 4, this step modifies the standard DETR target detection model to adapt it to the flag leaf water content regression task. The main modifications include: (1) removing the original target detection head (class prediction + bounding box regression); (2) replacing it with a water content regression prediction head; and (3) introducing a learnable regression query vector to replace the target query vector in DETR.

[0345] Encoder section:

[0346] In this embodiment, the input to the Transformer encoder is the multimodal fusion feature generated in step 4. Specifically:

[0347] If strategy B (concatenation and memory token attention) is adopted, the fused feature is fused_memory with shape (num_memory_tokens, B, 256), where num_memory_tokens = 4.

[0348] The encoder performs self-attention modeling on the fused features, and the output shape remains unchanged at (4, B, 256). This output serves as the memory feature map for the decoder.

[0349] In another embodiment, if strategy B is not used, the global fusion feature F_fusion (B, 256) output in step 4 can be directly expanded to (1, B, 256) and then input into the encoder.

[0350] Learnable regression query vector:

[0351] This embodiment introduces a learnable regression query vector to decode water content-related features from the memory feature map. Specific parameters are as follows:

[0352] Quantity: num_queries = 16 (default);

[0353] Dimensions: Each query vector has 256 dimensions;

[0354] Shape: (16, B, 256);

[0355] Initialization: Initialized using a truncated normal distribution (standard deviation 0.02);

[0356] Function: These query vectors interact with each other in the decoder through self-attention and aggregate information from the memory feature map through cross-attention.

[0357] Decoder structure:

[0358] The decoder uses the standard Transformer decoder architecture, with a total of 6 stacked layers (dec_layers = 6). Each layer contains the following sublayers:

[0359] Self-attention layer: Query vectors perform multi-head self-attention computations to capture the dependencies between different queries.

[0360] Cross-attention layer: Using the query vector as the Query and the memory feature map output by the encoder as the Key and Value, multi-head cross-attention calculation is performed to enable the query vector to extract water content-related information from the fused features.

[0361] Feedforward network layer: The structure is the same as FFNBlock in step 1 (Linear(256, 1024) → ReLU → Dropout(0.1) → Linear(1024, 256) → Dropout(0.1)).

[0362] Residual connections and layer normalization: Each sub-layer is followed by residual connections and layer normalization (using a pre-normalization architecture).

[0363] Regression prediction head:

[0364] The feature vector output by the decoder has a shape of (16, B, 256). Since this embodiment only needs to predict a single water content value, the 16 query outputs need to be aggregated into a single predicted value. Specifically, one of the following two methods is used:

[0365] Method 1 (default): Calculate the mean of the 16 query outputs along the query dimension to obtain (B, 256), and then input it into the regression prediction header.

[0366] Method 2: Flatten the 16 query outputs into (B, 16 × 256) = (B, 4096), and then map them to 1 dimension through a linear layer.

[0367] The regression prediction head employs a multilayer perceptron structure, mapping the aggregated features to the final predicted water content value. Its specific structure is as follows:

[0368] First layer: Linear layer, 256 dimensions → 256 dimensions, followed by GELU activation function and Dropout (0.1).

[0369] Second layer: Linear layer, 256 dimensions → 64 dimensions, followed by GELU activation function and Dropout (0.1).

[0370] The third layer: linear layer, 64-dimensional → 1-dimensional, outputs the predicted flag leaf water content (unit: percentage).

[0371] End-to-end prediction process:

[0372] The complete regression prediction process can be summarized as follows:

[0373] First, the multimodal fusion features generated in step 4 are used as a memory feature map and input into the decoder. Then, 16 learnable regression query vectors interact with the memory feature map through a 6-layer decoder (each layer contains self-attention, cross-attention, and feedforward networks) to extract water content-related information. Next, the 16 query vectors output by the decoder are aggregated into a single feature vector. Finally, the final flag leaf water content prediction value is mapped to the regression prediction head of a three-layer multilayer perceptron.

[0374] S6. Train the improved DETR regression model using the original dataset, and use the trained model to predict the slight loss of water content in highland barley plants.

[0375] The original dataset is divided into training, validation, and test sets in a 7:2:1 ratio and then input into the model for training.

[0376] We use a weighted combination of two loss functions, mean squared error and mean absolute error, as the model optimization objective to balance prediction bias and accuracy.

[0377] The AdamW optimizer is used to update the model parameters, combined with an early stopping strategy to prevent overfitting. R² (coefficient of determination), RMSE (root mean square error), and CORR (correlation coefficient) are used as the core evaluation indicators to iteratively optimize the model parameters until the model reaches the optimal performance on the validation set (R² ≥ 0.85, RMSE ≤ 5%).

[0378] After training is complete, the optimal model weights are saved for subsequent inference deployment.

[0379] S7. Model Validation and Performance Evaluation;

[0380] The trained model was independently validated using a test set. Hyperspectral and RGB dual-modal data that were not used in the training were input to obtain water content prediction results.

[0381] The model's R², RMSE, and MAE indices were calculated by comparing the actual moisture content calibrated by the drying method to verify the model's prediction accuracy, robustness, and generalization ability.

[0382] Specialized tests were conducted on barley samples at different growth stages and under different levels of drought stress to verify the model's adaptability in complex highland field conditions and ensure that the model meets actual testing needs.

[0383] S8, Model Lightweighting and System Optimization;

[0384] The trained model is lightweighted by using methods such as model pruning, parameter quantization, and knowledge distillation to compress the model size and reduce inference latency.

[0385] Export the model to ONNX format to adapt to the hardware environment of edge computing terminals and enable local inference of the model on field equipment;

[0386] Optimize the data processing flow and model inference logic to improve system operating efficiency and ensure that the entire process of sampling-collection-inference can be completed within 1 to 2 minutes, meeting the needs of rapid field detection.

[0387] The following explanation uses field online reasoning and water content prediction as an example.

[0388] After completing minimally invasive sampling, hyperspectral and RGB data acquisition in the field, the operators upload the data to the edge computing terminal.

[0389] The terminal automatically executes the preprocessing, feature extraction, fusion, and model inference processes from steps 2 to 5.

[0390] The model outputs the continuous water content prediction value (%) of the barley plant through the regression prediction head, thus completing the online prediction.

[0391] The terminal classifies the predicted water content into three water status levels: "normal, slightly water-deficient, and severely water-deficient" based on preset thresholds.

[0392] The terminal interface displays information such as plant number, collection time, water content, and water shortage level for operators to view;

[0393] The raw data, preprocessed features, and prediction results are stored in a local database for subsequent model iterations and field irrigation decisions.

[0394] The development and deployment of the entire system were completed, enabling minimal-damage, rapid, and accurate detection of the water content of barley plants in the Ganzi Tibetan Autonomous Prefecture, providing core data support for the intelligent irrigation system.

[0395] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for predicting the water content of barley plants with minimal loss based on DETR multimodal fusion, characterized in that, Includes the following steps: S1. Minimally damaged sampling and multimodal data acquisition of highland barley plants; Select barley plants to be tested and assign them unique numbers to ensure that subsequent data correspond one-to-one. For each numbered barley plant, remove the top healthy flag leaf with minimal damage to complete the minimally damaged sampling. The flag leaf was laid flat on the stage, and hyperspectral data of the flag leaf was collected under a standard light source to generate a continuous one-dimensional sequence of reflectance data from the visible light to the near-infrared band. The whole plant of the same barley plant was photographed to obtain two-dimensional RGB image data including plant shape, color and growth status; The flag leaf hyperspectral data collected according to the barley plant number was paired with RGB image data to obtain the original dataset containing flag leaf hyperspectral data, whole plant RGB data, and true water content labels. S2. Preprocess the flag leaf hyperspectral data and RGB image data in the original dataset; S3. Perform dual-modal feature extraction on the preprocessed data to extract one-dimensional spectral deep features related to water content inside the flag leaf and two-dimensional visual deep features related to water stress in the whole canopy. S4. Multimodal feature fusion based on DETR encoder; One-dimensional spectral deep features and two-dimensional visual deep features are simultaneously input into a multimodal feature fusion module based on a modified DETR encoder. Through the cross-attention mechanism of the multimodal feature fusion module, deep interaction and global modeling of one-dimensional spectral deep features and two-dimensional visual deep features are realized. One-dimensional spectral deep features supplement the details of water content inside the leaves, and two-dimensional visual deep features supplement the global information of the overall plant growth. After fusion, multimodal global features are output. S5. Construct an improved DETR regression model; The standard DETR target detection model is improved by removing the original target detection head and replacing it with a regression prediction structure, and introducing a learnable regression query vector to adapt to the water content regression task. The Transformer encoder and decoder structure is retained, and the decoder completes feature modeling through self-attention, cross-attention, and fully connected layers; A regression prediction head, i.e. a fully connected network, is connected after the decoder to map the fused multimodal global features into continuous numerical output, realizing end-to-end prediction from features to water content. S6. Train the improved DETR regression model using the original dataset, and use the trained model to predict the slight loss of water content in highland barley plants.

2. The method for predicting the water content of barley plants with minimal loss based on DETR multimodal fusion according to claim 1, characterized in that, Step S2 specifically includes: Flag leaf hyperspectral data preprocessing: Black and white correction, noise removal, outlier removal and reflectance normalization are performed on the original flag leaf hyperspectral data. The continuous projection algorithm is used to screen the feature bands that are highly correlated with the water content of barley, remove redundant information, and output a fixed-dimensional hyperspectral feature vector. RGB image data preprocessing: Perform size normalization, dehazing enhancement, background removal and pixel value normalization on the original RGB image data, unify the image input format, and eliminate lighting and background interference; Complete the time synchronization, spatial alignment and sample pairing of flag leaf hyperspectral data and RGB image data to ensure that the hyperspectral features and RGB features correspond to the same plant.

3. The method for predicting the water content of barley plants with minimal loss based on DETR multimodal fusion according to claim 1, characterized in that, Step S3 specifically includes: Flag leaf hyperspectral feature extraction: The preprocessed hyperspectral feature vector is encoded by an encoder of a fully connected network to extract one-dimensional deep spectral features related to water content inside the flag leaf. RGB visual feature extraction: The preprocessed RGB image is encoded using CNN and Transformer encoders to extract two-dimensional visual deep features of the whole canopy related to water stress.

4. The method for predicting the water content of highland barley plants with minimal loss based on DETR multimodal fusion according to claim 1, characterized in that, Step S6 specifically includes: The original dataset is divided into training set, validation set and test set according to a set ratio, and the model is trained accordingly. We use a weighted combination of mean squared error and mean absolute error as the model optimization objective to balance prediction bias and accuracy. The AdamW optimizer is used to update the model parameters, combined with an early stopping strategy to prevent overfitting. R², RMSE, and CORR are used as the core evaluation metrics to iteratively optimize the model parameters until the model reaches its optimal performance on the validation set. After training is completed, the optimal model weights are saved, and the water content of barley plants is predicted with minimal loss.

5. The method for predicting the water content of highland barley plants with minimal loss based on DETR multimodal fusion according to claim 4, characterized in that, The method also includes: S7. Model Validation and Performance Evaluation; The trained model was independently validated using a test set. Hyperspectral and RGB dual-modal data that were not used in the training were input to obtain water content prediction results. The model's R², RMSE, and MAE indices were calculated by comparing the actual moisture content calibrated by the drying method to verify the model's prediction accuracy, robustness, and generalization ability. Specialized tests were conducted on barley samples at different growth stages and with different levels of drought stress to verify the model's adaptability in complex highland field scenarios and ensure that the model meets actual testing needs.

Images