Plant salt tolerance response modeling prediction method and system based on time sequence image

Abstract

The invention relates to the technical field of image segmentation, in particular to a plant salt tolerance response modeling prediction method and system based on a time sequence image. The method comprises the following steps: preprocessing acquired plant sample image data; performing image segmentation on the preprocessed image data by using a plant semantic segmentation model based on U-Net; a plant salt tolerance response prediction model based on TimeSform is constructed; and predicting the plant salt tolerance response grade by using the plant salt tolerance response prediction model. According to the time sequence image-based plant salt tolerance response prediction modeling method provided by the invention, a prediction process integrating image acquisition, preprocessing, dynamic feature extraction and depth time sequence modeling is constructed, so that the efficiency, precision and automation level of plant salt tolerance phenotype recognition are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a plant salt tolerance response modeling and prediction method and system based on time series images. Background Art

[0002] The changes in plant growth under salt stress conditions are a dynamic and continuous physiological response process, typically manifested as restricted root growth, yellowing leaves, leaf margin dryness, plant dwarfing, and even death. Traditional methods for assessing plant salt tolerance rely primarily on regular manual observation and sampling to measure physiological indicators (such as SOD activity, chlorophyll content, and electrical conductivity). These methods are time-consuming, subject to significant human error, and inefficient, making them difficult to support large-scale salt tolerance screening or precise stress response research.

[0003] With the advancement of image processing and artificial intelligence technologies, extracting plant phenotypic traits using computer vision has become a key area of ​​plant phenomics. However, most existing studies focus on analyzing static images, which struggles to fully capture the continuous responses of plants to stress at different time points. In reality, plant salt tolerance exhibits significant temporal and process-dependent responses. Only by considering the dynamic trajectory of plant growth can a more scientific and accurate assessment of salt tolerance be achieved.

[0004] Therefore, constructing a plant salt tolerance response prediction modeling method based on time-series images can automatically extract plant stress response trends from "multi-time point image sequences" and build a deep learning model that can predict subsequent salt tolerance status. This is of great significance to promoting intelligent breeding screening, high-throughput stress phenotype identification and plant adversity physiology research. Summary of the Invention

[0005] In order to solve the above-mentioned problems, the present invention provides a plant salt tolerance response modeling and prediction method and system based on time series images.

[0006] In a first aspect, the present invention provides a plant salt tolerance response modeling and prediction method based on time series images, which adopts the following technical solutions: A plant salt tolerance response modeling and prediction method based on time series images, comprising: Acquiring plant sample image data; Preprocessing the acquired plant sample image data; Use the U-Net-based plant semantic segmentation model to perform image segmentation on the preprocessed image data; Construct a prediction model of plant salt tolerance response based on TimeSformer; The plant salt tolerance response prediction model is used to predict the plant salt tolerance response level.

[0007] Furthermore, the acquired plant sample image data is preprocessed, including size unification, brightness normalization, and format standardization of the collected images. All images are cropped to a rectangular area containing the complete plant body and uniformly scaled to a fixed size of 384×384 for batch input into the deep learning model. The adaptive histogram equalization (CLAHE) method is applied to enhance brightness and eliminate color deviation caused by illumination differences. The normalization formula is: , in, is the original image pixel, is the overall grayscale mean of the image, is the standard deviation; at the same time, a high-quality training dataset is constructed. The dataset includes the original plant image and its corresponding segmentation mask. The mask image is a binary image, where the plant pixel is assigned a value of 1 and the background is 0. Each pair of image-mask samples in the training set is: , in, For the i-th input image data, The corresponding mask.

[0008] Furthermore, the plant semantic segmentation model based on U-Net is used to perform image segmentation on the preprocessed image data, including using the U-Net encoder to extract the spatial hierarchical features of the image layer by layer, and converting the original input image into a feature map with rich semantic information but gradually reduced resolution. The encoder is composed of N repeated convolution blocks, each block contains two concatenated convolution layers, ReLU activation function and a 2×2 maximum pooling layer. After each layer of encoding, the size of the feature map is halved and the number of channels is doubled. Suppose the input image is , No. The feature map extracted by the layer encoder is , the expression of each convolution operation is: , in, Indicates the The convolution kernel of the layer, As a bias term, conditional label embedding is introduced to encode experimental meta-information including time point t, salt concentration c, variety type g into a conditional vector e, and the model feature distribution is dynamically adjusted through the channel modulation mechanism. Suppose the output feature of a certain convolutional layer is F, and the modulation method is: , in, are the channel scaling and translation factors computed by the fully connected layer from the conditional vector e.

[0009] Furthermore, the image segmentation of the pre-processed image data using the U-Net-based plant semantic segmentation model also includes introducing a temporal attention mechanism to enhance the model's ability to perceive temporal trends by fusing the semantic features of the current frame and adjacent frames in the encoding stage, wherein the current frame feature is assumed to be , and its preceding and following frames are 、 , the temporal attention module is used for all frames Perform weighted fusion of previous and next frames to generate new frames to fuse the information of temporally adjacent frames, thereby improving the temporal consistency of the segmentation results and their sensitivity to coercion trends. The fusion method is expressed as: , in, It is a learnable weight parameter generated by the self-attention mechanism. The specific generation method is: first, perform a global pooling operation on the current frame and the previous and next frames, compress the spatial dimension, and only retain the channel information. Then, the vectors of the three channels are spliced ​​into a matrix , is the vector of the corresponding time frame after global pooling, and then the matrix Perform linear transformations to calculate query, key, and value vectors respectively, and then calculate the attention weight matrix.

[0010] Furthermore, the use of the U-Net-based plant semantic segmentation model to perform image segmentation on the preprocessed image data also includes using the U-Net decoder to gradually restore the spatial resolution of the feature map, and restore the deep semantic information extracted by the encoder to a pixel-level prediction result of the same size as the original image, wherein the decoder is set to The input feature map of the layer is , the corresponding encoder The characteristics of the layer are , the processing process of the decoder is: first, the feature map is upsampled to enlarge the upper feature map, which is expressed as: ; The upsampled feature map is then concatenated with the shallow features of the encoder in the channel dimension to fuse local details with global semantic information, expressed as: , Finally, two convolution operations are performed to refine the fusion features and enhance spatial alignment and edge expression, which can be expressed as: , Through this restoration method, the decoder can restore the image structure step by step while effectively compensating for the spatial detail information lost in the encoding stage.

[0011] Furthermore, the image segmentation of the pre-processed image data using the U-Net-based plant semantic segmentation model also includes training the U-Net model using a multi-objective joint optimization strategy, wherein binary cross entropy (BCE) is used as a basic pixel-level supervision term to measure the accuracy of the model in classifying plant foreground and background, which is expressed as: , in, is the true value, is the predicted value; at the same time, the Dice loss function is introduced to process plant images with sparse foreground pixels, which is expressed as: , in is a smoothing term, and then a temporal consistency loss term is introduced to encourage the smooth change of the prediction results of adjacent frames, which is expressed as: , The final overall loss function is a weighted combination of three items: , in It is a parameter that balances the various loss functions.

[0012] Furthermore, the construction of a plant salt tolerance response prediction model based on TimeSformer includes using TimeSformer to divide each frame image into N image patches, mapping them into high-dimensional spatial embedding vectors through linear transformation to form an initial embedding tensor; and then superimposing spatial position encoding on each patch. Temporal Position Encoding , the final embedding vector is expressed as: , The TimeSformer consists of multiple stacked temporal Transformer encoding layers, each of which performs spatial attention and temporal attention in sequence, where the input of spatial attention is the patch features of each frame image. , query, key and value are obtained through learnable linear transformation; after spatial attention, temporal attention modeling is performed on the representation of each patch on the time axis. After all patches and frames are processed, sequence-level features are extracted from the overall feature sequence through global average pooling: , in, Characterize the stress response behavior of the entire plant throughout the entire time series. Subsequently, this feature is fed into the multi-layer perceptron (MLP) predictor to output the final response level: .

[0013] Furthermore, the method of predicting the plant salt tolerance response level using the plant salt tolerance response prediction model also includes adding a channel attention module after the input embedding of TimeSformer to enhance the model's selective attention to the feature channel dimension. Specifically, based on the SE attention mechanism, a channel attention extraction branch that integrates position information is added. For any feature map after the convolution operation, feature compression is performed along the height and width directions respectively, fully retaining the position information of the feature in different spatial dimensions, which is expressed as: , The two feature matrices are then deformed and concatenated, and the convolution operation is used to perform feature fusion and position relationship modeling, which can be expressed as: , in, is the weight matrix, Used to reduce feature dimensions and reduce computational overhead, is the ReLU activation function, followed by Separated into and , using the other two The convolutional layer will and The number of channels is restored, and the final position channel attention Expressed as: , in, is the Sigmod activation function, and is a split operation, and There are two weight matrices. The final module attention consists of two channels: SE attention and position attention. All attention information is fed back to the original feature map. After the convolution operation, a new feature map with added attention is obtained as the output. The update mechanism of the attention module is specifically expressed as follows: , in, is the output of this layer, is the ReLU activation function, is the input of this layer.

[0014] Furthermore, the method of predicting the plant salt tolerance response level using the plant salt tolerance response prediction model also includes training the plant salt tolerance response prediction model. During training, each sample input is a plant image sequence segmented by U-Net. After Patch Embedding, position encoding and channel attention extraction, it is sent to TimeSformer to extract global phenotypic response features Classification and prediction of plant salt tolerance levels , where the loss function selects the cross entropy loss function: ; At the same time, in order to maintain the consistency between output sequences, the inter-frame prediction stability regularization term is introduced: , The final total loss function is a weighted combination: .

[0015] In the second aspect, a plant salt tolerance response modeling and prediction system based on time series images includes: The data acquisition module is configured to acquire medical images and plant sample image data; A preprocessing module is configured to preprocess the acquired plant sample image data; a segmentation module configured to perform image segmentation on the preprocessed image data using a U-Net-based plant semantic segmentation model; The training module is configured to build a plant salt tolerance response prediction model based on TimeSformer; The prediction module is configured to predict the plant salt tolerance response level using the plant salt tolerance response prediction model.

[0016] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a plant salt tolerance response modeling and prediction method based on time series images.

[0017] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to implement the plant salt tolerance response modeling and prediction method based on time series images.

[0018] In summary, the present invention has the following beneficial technical effects: The proposed method for predicting plant salt tolerance responses based on time-series images significantly improves the efficiency, accuracy, and automation of identifying plant salt tolerance phenotypes by building a prediction process that integrates image acquisition, preprocessing, dynamic feature extraction, and deep time-series modeling. Compared to traditional manual observation and single-point image analysis methods, this method offers the following specific benefits: Traditional manual measurement takes 2–5 minutes per plant, while this method processes a single image in less than 10 seconds, enabling daily processing of image data for over 2,000 plants, significantly improving screening efficiency. The CNN-LSTM model achieves a 92.3% accuracy rate for predicting plant salt tolerance, and a 0.91 R² regression value for root length prediction, outperforming static models. Tests on Arabidopsis thaliana, foxtail millet, and alfalfa revealed an accuracy rate exceeding 85%, demonstrating cross-species adaptability. The correlation between image feature predictions and physiological data such as SOD, Pro, and chlorophyll reached over 0.83, enabling it to replace some destructive detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of a plant salt tolerance response modeling and prediction method based on time series images according to Example 1 of the present invention.

[0020] Figure 2 This is a comparison diagram of segmentation results of a plant salt tolerance response modeling and prediction method based on time series images in Example 1 of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] Example 1 Reference Figure 1 The present embodiment provides a plant salt tolerance response modeling and prediction method based on time series images, including: (1) Plant sample processing and image acquisition 1) Material preparation and sowing process Select model or target plant materials such as Arabidopsis thaliana, Setaria officinalis, alfalfa, or millet, sterilize them according to standard procedures, and then sow them. Treat them with 70% ethanol for 1 minute, rinse them, and then treat them in 1% sodium hypochlorite for 5 minutes, then rinse them three times with sterile water. Use 1 / 2 MS culture medium or seedling substrate soil. The sowing density is controlled at , where D is the sowing density per unit area, N is the number of seeds per dish, and A is the effective sowing area of ​​the dish.

[0023] 2) Salt stress treatment plan design After plant emergence, stress treatment was performed using NaCl solution concentrations of 0, 25, 50, 75, 100, 125, and 150 mM to simulate different stress intensities. Root irrigation was used daily to irrigate the roots with NaCl solution of the corresponding concentration. The stress intensity was also labeled for subsequent supervised model learning. The specific representation of the stress intensity label quantification is: , in, is the standardized salt stress level of treatment group i, NaCl concentration of group i, is the control group concentration, is the maximum NaCl concentration. The supervised labels of the image sequence are fed into the model for training.

[0024] 3) Time-series image acquisition design To model the temporal dynamics of plant salt tolerance, this solution designed a timed image acquisition process to construct multi-timepoint image sequences. Specific capture areas included top views for observing leaf area and color changes; side views for observing plant height, growth potential, and leaf curl; and root system maps for detecting changes in root length and density. All images were captured using a fixed-angle, uniform light source and a white PVC background.

[0025] (2) Image preprocessing To ensure consistency in image input across time points and samples, the captured images were resized, brightness normalized, and format standardized. All images were cropped to a rectangular area encompassing the entire plant body and rescaled to a fixed size of 384×384 for batch input into the deep learning model. Furthermore, methods such as adaptive histogram equalization (CLAHE) were applied for brightness enhancement to eliminate color deviations caused by illumination differences. The normalization formula is as follows: , in, is the original image pixel, is the overall grayscale mean of the image, is the standard deviation.

[0026] In order to achieve the subsequent semantic segmentation of plants, it is necessary to build a high-quality training dataset. The dataset includes the original plant image and its corresponding segmentation mask. The mask image is a binary image, where the plant pixel is assigned a value of 1 and the background is assigned a value of 0. Each image-mask pair in the training set is: , in, For the i-th input image data, The corresponding mask.

[0027] (3) Plant semantic segmentation model based on U-Net Semantic segmentation technology is a key basic link in realizing temporal modeling of plant images. Its role is to accurately extract plant areas from complex backgrounds and provide high-quality, structurally consistent mask images as input for subsequent feature extraction and temporal modeling. By performing pixel-by-pixel segmentation on structures such as plant leaves and stems, semantic segmentation can accurately quantify changes in plant morphology, color, and edges, especially when phenotypic degradation processes such as yellowing, curling, and withering occur under salt stress conditions, it can improve the stability and accuracy of image phenotypic perception. The present invention adopts a U-Net plant semantic segmentation network based on a temporal attention mechanism, combined with temporal attention and conditional embedding mechanisms, so that the segmentation results maintain continuity in the time dimension. U-Net is a typical encoder-decoder (Encoder-Decoder) architecture semantic segmentation neural network, and its specific structure is divided into encoder and decoder: 1) Encoder The U-Net encoder is the downsampling path in the network structure. Its function is to extract the spatial hierarchical features of the image layer by layer, converting the original input image into a feature map with rich semantic information but gradually decreasing resolution. The encoder consists of N repeated convolution blocks, each of which contains two concatenated convolution layers (3×3 convolution kernels), a ReLU activation function, and a 2×2 max pooling layer to reduce the spatial size. After each encoding layer, the size of the feature map is halved and the number of channels is doubled. Suppose the input image is , No. The feature map extracted by the layer encoder is , the expression of each convolution operation is: , in, Indicates the The convolution kernel of the layer, is the bias term.

[0028] Plant response patterns vary significantly under varying salt concentrations, treatment durations, and genotypes. Standard U-Net models are unable to explicitly perceive these treatment contexts, resulting in missegmentation of similar images under different conditions. This leads to poor generalization, particularly in multi-material, multi-gradient treatment experiments. To address this, the present invention introduces a conditional label embedding module that encodes experimental metadata, including time point t, salt concentration c, and variety type g, into a conditional vector e. This module then dynamically adjusts the model's feature distribution through a channel modulation mechanism. Let the output feature of a convolutional layer be F, and the modulation scheme be as follows: , in, are the channel scaling and translation factors computed by the fully connected layer from the conditional vector e.

[0029] In the analysis of plant salt tolerance response, the temporal nature of images is of great significance. A single image is difficult to reflect the subtle changes in plants during stress, especially trend phenotypes such as early yellowing and leaf drooping. The standard U-Net model processes each frame of the image independently, ignoring the continuity between adjacent time frames, resulting in unstable segmentation results and obvious jumps between upper and lower frames. To this end, the present invention introduces a temporal attention module to enhance the model's perception of temporal change trends by fusing the semantic features of the current frame and adjacent frames in the encoding stage. Suppose the current frame feature is , and its preceding and following frames are 、 , the temporal attention module is used for all frames Perform weighted fusion of previous and next frames to generate new frames to integrate information from temporally adjacent frames, thereby improving the temporal consistency of the segmentation results and their sensitivity to coercion trends. The specific fusion method is as follows: , in, It is a learnable weight parameter generated by the self-attention mechanism. The specific generation method is: first, perform a global pooling operation on the current frame and the previous and next frames, compress the spatial dimension, and only retain the channel information. Then, the vectors of the three channels are spliced ​​into a matrix , is the vector of the corresponding time frame after global pooling. Then use the matrix Perform linear transformations and calculate the query, key, and value vectors respectively: , in, are learnable weights, It is the attention dimension.

[0030] Then calculate the attention weight matrix : , Here, the value of row 2 represents the attention weight of the current frame t for the jth frame, that is: , During the encoding process of the original image, the encoder extracts features from the shallowest to the deepest layers. The bottom-level features extract edge, color, and texture information, such as leaf greenness, spots, and wilting edges; the middle-level features integrate structural morphological information, such as leaf size and arrangement; and the deep-level features focus on high-level semantic information, such as the overall plant outline and growth status. These features are passed to the decoder via skip connections, effectively aggregating features at different levels and improving feature discrimination capabilities.

[0031] 2) Decoder The decoder of U-Net is the upsampling path part in its network structure. Its core function is to gradually restore the spatial resolution of the feature map and restore the deep semantic information extracted by the encoder to a pixel-level prediction result of the same size as the original image. For semantic segmentation tasks, the quality of the decoder directly determines the clarity of the segmentation boundary, the degree of contour restoration, and the integrity of the target structure. Especially in plant image processing, the ability to restore details such as slender leaves, jagged edges, and dry spots is crucial. The decoder consists of multiple upsampling modules, each of which includes an upsampling operation, a feature concatenation operation (concatenation) with a skip connection to the corresponding encoder layer, and two consecutive 3×3 convolutional layers and ReLU activation functions. The main purpose of the upsampling operation is to enlarge the low-resolution, high-semantic feature map to a higher spatial size, so that it gradually approaches the resolution of the input image. Suppose the decoder is The input feature map of the layer is , the corresponding encoder The characteristics of the layer are , the decoder processing process is: 1. Upsampling The feature map is amplified by upsampling the upper feature map, which is specifically expressed as: , 2. Skip Connection Splicing The upsampled feature map is concatenated with the shallow features of the encoder in the channel dimension to fuse local details with global semantic information. Specifically expressed as: , 3. Convolutional recovery Through two convolution operations, the fusion features are refined and the spatial alignment and edge expression are enhanced. The specific expression is as follows: , This restoration method enables the decoder to gradually restore the image structure while effectively compensating for spatial detail lost during the encoding phase. Decoder performance is particularly critical for plant images. For example, when leaf edges are blurred and color changes unevenly after salt stress, skip connections can restore leaf edge details, enhancing the model's ability to detect phenotypes such as yellowing and curling.

[0032] 3) Loss function and training method In the predictive modeling of plant salt tolerance response based on time series images, the training process of the semantic segmentation model must not only ensure the accurate segmentation of single-frame images, but also the consistency of the model in the temporal dimension and its adaptability to different processing conditions. Therefore, the improved model TAC-U-Net proposed in this patent adopts a multi-objective joint optimization strategy in the design of loss function and training method, combining traditional pixel-level segmentation accuracy indicators with regularization terms that enhance the temporal robustness and conditional adaptability of the model to ensure the overall performance of the model under multi-dimensional objectives. First, binary cross entropy (BCE) is used as the basic pixel-level supervision term to measure the accuracy of the model's classification of plant foreground and background. The formula is: , in, is the true value, is the predicted value.

[0033] Secondly, to improve the model's overall grasp of the target area, the Dice loss function is introduced to process plant images with sparse foreground pixels. The Dice loss can be expressed as: , in is the smoothing term.

[0034] In addition, to ensure that the model has temporal continuity when processing image sequences, a temporal consistency loss term is introduced to encourage smooth changes in the prediction results of adjacent frames, which is defined as: , Ultimately, the overall loss function is a weighted combination of three terms: , in It is a parameter that balances the various loss functions.

[0035] The final training parameters of the network are , achieving a good balance between accuracy and consistency. Regarding training, this method employs a three-frame collaborative input mechanism, feeding the current frame and adjacent frames into the network for temporal attention fusion. The model only supervises the current frame, maintaining training stability. Furthermore, each sample image is accompanied by a conditional label, embedded as a vector and injected into the model via a modulation layer, effectively improving the model's adaptability to experimental treatment variations. The hyperparameter settings of TAC-U-Net combine the characteristics of the plant salt stress image segmentation task with the requirements of temporal consistency modeling, ensuring segmentation accuracy while balancing training stability and computational efficiency. The model input consists of a normalized RGB image and the corresponding conditional label embeddings. The encoder layer utilizes a four-layer convolutional downsampling architecture. Each layer has a 3×3 kernel size, a stride of 1, and padding of 1. The activation function is ReLU, and batch normalization is added after each layer to accelerate convergence. The decoder layer uses transposed convolution for upsampling and integrates encoder features via skip connections. In the temporal attention module, the time window size is set to 3 frames, and a multi-head self-attention mechanism is used with 4 attention heads and a channel embedding dimension of 64. The conditional label embedding layer has a dimension of 16 and is applied to the intermediate feature layer through conditional FiLM. The optimizer uses AdamW, and the initial learning rate is set to 1×10⁻. 4 , weight decay is 1×10⁻ 5 , the batch size is 8, and the maximum number of training rounds (Epoch) is 200.

[0036] 4) Segmentation image labeling After using U-Net to perform semantic segmentation on plant images, the segmentation results need to be further processed and labeled to construct a standardized training dataset or annotated file for the model's prediction output. In this method, the U-Net model outputs a single-channel mask image (mask) of the same size as the original image. Each pixel value is a probability value between [0, 1], indicating the confidence that the pixel belongs to the plant foreground. First, this probability mask is thresholded, marking regions with pixel values ​​greater than or equal to the threshold as foreground (value 1) and the rest as background (value 0), resulting in a clear binary mask image. Then, a contour extraction algorithm is used to identify and extract the plant region's shape, deriving basic phenotypic features such as leaf boundaries, plant height, and area. Furthermore, to support temporal modeling of multi-time point images, the system associates the mask image with information such as the original image's timestamp, treatment concentration, and sample number, generating a one-to-one image label file. The label file uses a standard CSV file format. Each record contains the image file name, time point, treatment group, leaf area, plant height, root length, and other indicator values. In addition, to construct regression-based supervision task labels, plant physiological data and image mask results are combined to generate hierarchical labels or continuous response values, achieving numerical coupling between images and salt tolerance phenotypes. Physiological indicators used include: SOD activity, electrical conductivity, MDA content, and chlorophyll content. Label generation is as follows: , in, is the set weight, is the image feature score of the i-th sample, which is obtained based on the yellow flower area and leaf area change rate of the plant in the image. Respectively represent the SOD activity, conductivity, MDA content, and chlorophyll content after normalization. The final stress response index is , with 0.2 as the dividing line, it is divided into 5 levels, indicating its salt tolerance. This label is also the classification target of the subsequent image time series prediction model.

[0037] (4) Plant salt tolerance response prediction model based on TimeSformer 1) Model structure In the task of predictive modeling of plant salt tolerance responses, images not only represent static morphological information but also a dynamic phenotypic process with temporal continuity. Under salt stress, plants undergo gradual changes from normal appearance to mild yellowing, leaf curling, and severe withering. These subtle physiological and morphological evolutions are often presented in a continuous, gradual manner in the image time series. Therefore, traditional static models, based solely on single-frame image analysis, have difficulty capturing the plant's evolutionary trends and response rhythms in the temporal dimension, and are prone to overlooking the hysteresis effects and weak manifestations of stress accumulation over a short period of time. To this end, it is crucial to introduce a structure with powerful temporal modeling capabilities and the ability to jointly perceive spatial and temporal features.

[0038] TimeSformer (Time-Space Transformer) is a visual Transformer structure designed for video time series modeling tasks. It combines the spatial perception of local image details with the ability to model global dependencies between frame sequences, making it very suitable for salt tolerance response analysis tasks in plant time series images. Compared with traditional time series neural networks such as LSTM and GRU, TimeSformer can capture both long-term and short-term phenotypic change trends, does not rely on strict sequence order input, and is highly robust to uneven changes between frames. In addition, compared to 3D convolutional models, TimeSformer can model the time dimension more efficiently and supports flexible adjustment of the number of frames, making it the preferred choice in high-throughput image processing scenarios. The input of TimeSformer is a plant image sequence, denoted as: , Where T is the number of time frames. Each frame image I has been processed by the plant semantic segmentation model described above, retaining only the plant region. To facilitate processing by the Transformer model, TimeSformer first divides each frame into fixed-size image patches (N patches per frame). Each patch is flattened into a vector and mapped into a high-dimensional embedding vector through a linear transformation, forming the initial embedding tensor: , Where D is the feature dimension. In order to make the model have the ability to perceive spatial and temporal positions, each patch is superimposed with a spatial position encoding. Temporal Position Encoding , the final embedding vector is expressed as: , The main structure of TimeSformer consists of multiple stacked temporal Transformer encoding layers. Each layer sequentially executes two attention modules: a spatial attention module that processes the relationships between patches within each frame and a temporal attention module that models the evolution of the same patch over time. The spatial attention module is processed as follows: , Among them, the input is the patch feature of each frame image , we get the query, key, and value through a learnable linear transformation: Spatial attention focuses on extracting the structural relationships between different leaf regions in the image, such as the spatial differences between dry and healthy regions. Subsequently, temporal attention modeling is performed on the representation of each patch on the time axis after spatial attention. Let the features of the patch at position 𝑛 in each frame be: , Temporal attention is calculated as: , in, The temporal attention module can be understood as focusing on the "phenotypic changes of the same leaf region over time," identifying trends such as yellowing and shrinkage. To stabilize training and increase expressive power, each layer of spatial and temporal attention is followed by a LayerNorm layer and a feed-forward network (FFN): , Finally, after all patches and frames are processed, the model extracts sequence-level features from the overall feature sequence through global average pooling: , in, Characterize the stress response behavior of the entire plant throughout the entire time series. This feature is then fed into a multi-layer perceptron (MLP) predictor to output the final response level: , When applying TimeSformer to predictive modeling of plant salt tolerance, although its native architecture possesses excellent temporal modeling and spatial perception capabilities, given the sparse distribution of phenotypic changes in plant time-series images, TimeSformer cannot effectively extract dependencies between different channels when processing plant image sequences. This results in insufficient representation of physiological characteristics such as subtle color changes and structural decay. To address this issue, this patent improves TimeSformer by adding a channel attention module after the input embedding of TimeSformer to enhance the model's selective attention to feature channel dimensions.

[0039] Attention mechanism is widely used in various neural networks to improve network performance. Channel attention mechanism is a form of attention used to establish attention relationships between features. Channel attention methods represented by SE attention mechanism effectively improve the performance of CNN network for image processing tasks. The core idea of ​​SE attention is compression and excitation. The compression operation is in any convolution operation. Afterwards, the spatial information of the entire feature map is compressed and aggregated into a single channel feature. This compression method can be implemented using global average pooling. For TimeSformer, this channel attention is specifically expressed as: , in, and are the time length and number of patches of the input feature map. Compressed to a single feature dimension Used for subsequent attention generation.

[0040] The SE attention mechanism uses the excitation operation to model the channel information obtained by the compression operation and extract the attention weights between channels. The excitation operation first reduces the feature dimension through the fully connected layer. times to compress channel information; then, the feature is transformed through a nonlinear activation function, and then another fully connected layer is used to restore the feature dimension to the original number of channels; finally, the sigmoid function is used to obtain the normalized weight coefficient, which is expanded to the shape of the original input to obtain the attention between channels. This channel attention can be expressed as: , in, is the Sigmoid function, is the ReLU activation function, and is the weight matrix.

[0041] Although this channel attention mechanism can effectively explore the dependencies between features and thus improve the expressive power of the network, it has the problem of ignoring the feature position information. Position information is of great significance for guiding the spatial distribution of attention. To this end, this patent has made improvements based on the SE attention module and designed a channel attention module LSE that can fuse position information. Based on the SE attention mechanism, this module adds a channel attention extraction branch that fuses position information. The main process of this branch is: for any feature map after convolution operation, feature compression is performed along the height and width directions respectively, fully retaining the position information of the features in different spatial dimensions. This compression method can be implemented using global average pooling, which is specifically expressed as: , In order to further capture the spatial position information in the features, this patent deforms and splices the two feature matrices and uses convolution operations to perform feature fusion and position relationship modeling. This process can be expressed as: , in, is the weight matrix, Used to reduce feature dimensions and reduce computational overhead, is the ReLU activation function, followed by Separated into and , using the other two The convolutional layer will and The number of channels is restored, and the final position channel attention Expressed as: , in, is the Sigmod activation function, and is a split operation, and are two weight matrices.

[0042] The final module attention consists of two channels: SE attention and position attention. Finally, all attention information is fed back to the original feature map, and a new feature map with added attention is obtained as output after convolution operation. The update mechanism of this attention module is specifically expressed as follows: , in, is the output of this layer, is the ReLU activation function, is the input of this layer.

[0043] 2) Model training The TimeSformer training method must fully consider practical issues such as temporal continuity, subtle phenotypic changes, and limited sample size in predictive modeling of plant salt tolerance. Therefore, a strategy combining frame-level enhancement, temporal regularization, and conditional control is adopted to improve model stability and generalization capabilities. During training, each sample input is a sequence of plant images segmented by U-Net. After Patch Embedding, position encoding and channel attention extraction, it is sent to TimeSformer to extract global phenotypic response features Classification and prediction of plant salt tolerance levels , where the loss function selects the cross entropy loss function: , In addition, in order to maintain the consistency between output sequences, the inter-frame prediction stability regularization term is introduced: , The final total loss function is a weighted combination: , The hyperparameter design of the improved TimeSformer combines the temporal characteristics of plant salt tolerance prediction with the requirements of multi-task supervision, aiming to enhance the ability to capture subtle phenotypic changes while maintaining global spatiotemporal modeling capabilities. The model input is a sequence of RGB plant foreground images segmented and normalized by TAC-U-Net, with a temporal frame count of 8 (covering key time points during the complete salt stress process). Video patches are segmented to a size of 16×16, with each frame containing 196 patches. Position encoding uses a spatial and temporal separation, with a channel attention module added after each spatial and temporal attention module, with a channel count of 768. The total network depth is 12 Transformer Block layers, the multi-head self-attention mechanism has 12 attention heads, each with a dimension of 64, a hidden layer dimension (MLPhidden size) of 3072, and a dropout probability of 0.1. The conditional label embedding dimension is set to 32. The optimizer uses AdamW with an initial learning rate of 5×10⁻. 5 , weight decay 0.05, batch size (Batch Size) is 4, and the maximum number of training rounds (Epoch) is 100.

[0044] In summary, this method includes five core steps. First, a standardized experimental design is used to collect image time series of plants under different salt concentrations and treatment cycles, and the U-Net model is used to perform semantic segmentation on the images, extracting the foreground area of ​​the plants to remove background interference. Secondly, key image features such as leaf area and yellowing index are extracted from the segmentation results to generate image feature scores, and response labels are constructed in combination with physiological indicators. Then, the processed image sequence is input into the improved TimeSformer model to extract dynamic phenotypic change characteristics. Finally, through supervised training of the model, the salt tolerance response level of the plant is output, realizing the prediction and quantification of physiological phenotypic changes of plants under salt stress.

[0045] Table 1 Comparison of segmentation performance between this paper and other U-Net networks Model Name Intersection and Union Dice coefficient F1 indicator FPS U-Net 0.812 0.875 0.872 32 Swin-Unet 0.865 0.903 0.905 20 TAC-U-Net (this method) 0.865 0.914 0.911 29 As shown in Table 1, experiments were conducted based on a self-collected time-series image dataset of Bermuda grass salt tolerance. This dataset contains 1,280 plant samples across four salt concentration gradients and five treatment time points, each containing multiple consecutive frames and corresponding physiological indicators. Before entering the time-series model, all images were segmented using a U-Net to extract the plant foreground region, and manually annotated to ensure the accuracy of the segmentation labels. This patent evaluated the performance of several mainstream semantic segmentation methods, and the results showed that the proposed method demonstrated outstanding segmentation accuracy and overall performance. In terms of intersection-over-union (IoU), TAC-U-Net and Swin-Unet tied for the highest, both reaching 0.865, significantly outperforming U-Net. In terms of the Dice coefficient, TAC-U-Net achieved 0.914, an improvement of 1.1 percentage points over Swin-Unet and 3.9 percentage points over U-Net. In terms of the F1 metric, TAC-U-Net achieved 0.911, surpassing other methods, demonstrating a high balance between accuracy and recall in identifying foreground pixels. Although the inference speed of TAC-U-Net is 29 frames per second, slightly lower than the original U-Net, its inference speed can still meet the application requirements of actual high-throughput phenotyping analysis while ensuring high segmentation accuracy and temporal consistency. These results show that TAC-U-Net balances accuracy and usability on this dataset, especially in terms of fine boundary segmentation and stable performance under different salt treatment conditions. Figure 2 As shown in the figure, it is a comparison diagram of the segmentation results, where Figure 2 (a) in the figure represents the original anemone picture. Figure 2 (b) Picture of native water horsetooth, Figure 2 (c) and Figure 2 (d) in the figure represents the segmentation result of the original U-Net. Figure 2 (e) and Figure 2 (f) in the figure represents the segmentation result of Swin-Unet. Figure 2 (g) and Figure 2 (h) in the figure represents the segmentation result of this method. The segmentation result shows that the proposed TAC-U-Net can achieve more refined plant segmentation and provide support for subsequent time series prediction.

[0046] Table 2 Comparison of the prediction method in this paper with other methods Model Name Accuracy F1 Weighted-F1 Early accuracy ResNet50+MLP 74.2% 0.702 0.718 68.0% CNN+LSTM 78.5% 0.743 0.761 72.4% 3D-CNN 80.1% 0.755 0.773 74.9% Swin-Transformer 83.2% 0.781 0.792 77.1% TimeSformer 85.4% 0.807 0.816 80.2% Improve TimeSformer (this method) 87.9% 0.829 0.838 83.5% As shown in Table 2, performance evaluation results on the prediction set based on the segmented dataset demonstrated that our proposed method outperformed the comparison methods across all metrics. Specifically, in terms of accuracy, our proposed method achieved 87.9%, a 2.5 percentage point improvement over the original TimeSformer, significantly outperforming methods such as 3D-CNN, Swin-Transformer, and CNN+LSTM. In terms of the F1 metric, our proposed method achieved 0.829, a 2.2% improvement over the original TimeSformer. The Weighted-F1 metric also improved from 0.816 to 0.838, demonstrating balanced prediction capabilities across all categories. In terms of early prediction accuracy, our proposed method achieved 83.5%, a 3.3 percentage point improvement over the original TimeSformer, demonstrating its enhanced ability to identify trends and capture phenotypic changes in images from the early stages of salt stress. These results demonstrate that the improved TimeSformer maintains overall prediction accuracy while offering significant advantages, particularly in early-stage stress prediction scenarios, providing reliable technical support for rapid salt tolerance screening and high-throughput analysis.

[0047] Example 2 This embodiment provides a plant salt tolerance response modeling and prediction system based on time series images, including: The data acquisition module is configured to acquire medical images and plant sample image data; A preprocessing module is configured to preprocess the acquired plant sample image data; a segmentation module configured to perform image segmentation on the preprocessed image data using a U-Net-based plant semantic segmentation model; The training module is configured to build a plant salt tolerance response prediction model based on TimeSformer; The prediction module is configured to predict the plant salt tolerance response level using the plant salt tolerance response prediction model.

[0048] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a plant salt tolerance response modeling and prediction method based on time series images.

[0049] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded and executed by the processor to implement a plant salt tolerance response modeling and prediction method based on time series images.

[0050] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A plant salt tolerance response modeling and prediction method based on time series images, characterized in that: include: Acquiring plant sample image data; Preprocessing the acquired plant sample image data; Use the U-Net-based plant semantic segmentation model to perform image segmentation on the preprocessed image data; Construct a prediction model of plant salt tolerance response based on TimeSformer; The plant salt tolerance response prediction model is used to predict the plant salt tolerance response level.

2. A plant salt tolerance response modeling and prediction method based on time series images according to claim 1, characterized in that: The obtained plant sample image data is preprocessed, including size unification, brightness normalization, and format standardization of the collected images. All images are cropped to a rectangular area containing the complete plant body and uniformly scaled to a fixed size of 384×384 for batch input into the deep learning model. The adaptive histogram equalization (CLAHE) method is applied to enhance brightness and eliminate color deviation caused by illumination differences. The normalization formula is: ， in, is the original image pixel, is the overall grayscale mean of the image, is the standard deviation; at the same time, a high-quality training dataset is constructed. The dataset includes the original plant image and its corresponding segmentation mask. The mask image is a binary image, where the plant pixel is assigned a value of 1 and the background is 0. Each pair of image-mask samples in the training set is: ， in, For the i-th input image data, The corresponding mask.

3. A plant salt tolerance response modeling and prediction method based on time series images according to claim 2, characterized in that: The plant semantic segmentation model based on U-Net is used to perform image segmentation on the preprocessed image data, including using the U-Net encoder to extract the spatial hierarchical features of the image layer by layer, and converting the original input image into a feature map with rich semantic information but gradually decreasing resolution. The encoder consists of N repeated convolution blocks, each of which contains two concatenated convolution layers, a ReLU activation function and a 2×2 maximum pooling layer. After each layer of encoding, the size of the feature map is halved and the number of channels is doubled. Suppose the input image is , No. The feature map extracted by the layer encoder is , the expression of each convolution operation is: , in, Indicates the The convolution kernel of the layer, As a bias term, conditional label embedding is introduced to encode experimental meta-information including time point t, salt concentration c, variety type g into a conditional vector e, and the model feature distribution is dynamically adjusted through the channel modulation mechanism. Suppose the output feature of a certain convolutional layer is F, and the modulation method is: ， in, are the channel scaling and translation factors computed by the fully connected layer from the conditional vector e.

4. A plant salt tolerance response modeling and prediction method based on time series images according to claim 3, characterized in that: The U-Net-based plant semantic segmentation model is used to segment the pre-processed image data, and a temporal attention mechanism is introduced to enhance the model's perception of temporal change trends by fusing the semantic features of the current frame and adjacent frames in the encoding stage. , and its preceding and following frames are 、 , the temporal attention module is used for all frames Perform weighted fusion of previous and next frames to generate new frames to fuse the information of temporally adjacent frames, thereby improving the temporal consistency of the segmentation results and their sensitivity to coercion trends. The fusion method is expressed as: ， in, It is a learnable weight parameter generated by the self-attention mechanism. The specific generation method is: first, perform a global pooling operation on the current frame and the previous and next frames, compress the spatial dimension, and only retain the channel information. Then, the vectors of the three channels are spliced ​​into a matrix , is the vector of the corresponding time frame after global pooling, and then the matrix Perform linear transformations to calculate query, key, and value vectors respectively, and then calculate the attention weight matrix.

5. A plant salt tolerance response modeling and prediction method based on time series images according to claim 4, characterized in that: The method uses a U-Net-based plant semantic segmentation model to segment the pre-processed image data, and also includes using a U-Net decoder to gradually restore the spatial resolution of the feature map, and restore the deep semantic information extracted by the encoder to a pixel-level prediction result of the same size as the original image, wherein the decoder is set to The input feature map of the layer is , the corresponding encoder The characteristics of the layer are , the processing process of the decoder is: first, the feature map is upsampled to enlarge the upper feature map, which is expressed as: ； The upsampled feature map is then concatenated with the shallow features of the encoder in the channel dimension to fuse local details with global semantic information, expressed as: ， Finally, two convolution operations are performed to refine the fusion features and enhance spatial alignment and edge expression, which can be expressed as: ， Through this restoration method, the decoder can restore the image structure step by step while effectively compensating for the spatial detail information lost in the encoding stage.

6. A plant salt tolerance response modeling and prediction method based on time series images according to claim 5, characterized in that: The method of using a U-Net-based plant semantic segmentation model to segment the preprocessed image data also includes training the U-Net model using a multi-objective joint optimization strategy. Binary cross entropy (BCE) is used as a basic pixel-level supervision term to measure the accuracy of the model in classifying plant foreground and background, which is expressed as: ， in, is the true value, is the predicted value; at the same time, the Dice loss function is introduced to process plant images with sparse foreground pixels, which is expressed as: ， in is a smoothing term, and then a temporal consistency loss term is introduced to encourage the smooth change of the prediction results of adjacent frames, which is expressed as: ， The final overall loss function is a weighted combination of three items: ， in It is a parameter that balances the various loss functions.

7. A plant salt tolerance response modeling and prediction method based on time series images according to claim 6, characterized in that: The proposed plant salt tolerance response prediction model based on TimeSformer includes using TimeSformer to divide each frame image into N image patches, mapping them into high-dimensional spatial embedding vectors through linear transformation to form an initial embedding tensor; and then superimposing spatial position encoding on each patch. Temporal Position Encoding , the final embedding vector is expressed as: ， The TimeSformer consists of multiple stacked temporal Transformer encoding layers, each of which performs spatial attention and temporal attention in sequence, where the input of spatial attention is the patch features of each frame image. , query, key and value are obtained through learnable linear transformation; after spatial attention, temporal attention modeling is performed on the representation of each patch on the time axis. After all patches and frames are processed, sequence-level features are extracted from the overall feature sequence through global average pooling: ， in, Characterize the stress response behavior of the entire plant throughout the entire time series. Subsequently, this feature is fed into the multi-layer perceptron (MLP) predictor to output the final response level: .

8. A plant salt tolerance response modeling and prediction method based on time series images according to claim 7, characterized in that: The method of predicting the plant salt tolerance response level using the plant salt tolerance response prediction model also includes adding a channel attention module after the input embedding of TimeSformer to enhance the model's selective attention to the feature channel dimension. Specifically, based on the SE attention mechanism, a channel attention extraction branch that integrates position information is added. For any feature map after the convolution operation, feature compression is performed along the height and width directions respectively, fully retaining the position information of the feature in different spatial dimensions, which is expressed as: ， The two feature matrices are then deformed and concatenated, and the convolution operation is used to perform feature fusion and position relationship modeling, which can be expressed as: ， in, is the weight matrix, Used to reduce feature dimensions and reduce computational overhead, is the ReLU activation function, followed by Separated into and , using the other two The convolutional layer will and The number of channels is restored, and the final position channel attention Expressed as: ， in, is the Sigmod activation function, and is a split operation, and There are two weight matrices. The final module attention consists of two channels: SE attention and position attention. All attention information is fed back to the original feature map. After the convolution operation, a new feature map with added attention is obtained as the output. The update mechanism of the attention module is specifically expressed as follows: ， in, is the output of this layer, is the ReLU activation function, is the input of this layer.

9. The method for modeling and predicting plant salt tolerance response based on time series images according to claim 8, characterized in that: The method of using the plant salt tolerance response prediction model to predict the plant salt tolerance response level also includes training the plant salt tolerance response prediction model. During training, each sample input is a plant image sequence segmented by U-Net. After Patch Embedding, position encoding and channel attention extraction, it is sent to TimeSformer to extract global phenotypic response features Classification and prediction of plant salt tolerance levels , where the loss function selects the cross entropy loss function: At the same time, in order to maintain the consistency between the output sequences, the inter-frame prediction stability regularization term is introduced: ， The final total loss function is a weighted combination: .

10. A plant salt tolerance response modeling and prediction system based on time series images, characterized in that: include: A data acquisition module is configured to acquire medical images; Acquiring plant sample image data; A preprocessing module is configured to preprocess the acquired plant sample image data; a segmentation module configured to perform image segmentation on the preprocessed image data using a U-Net-based plant semantic segmentation model; The training module is configured to build a plant salt tolerance response prediction model based on TimeSformer; The prediction module is configured to predict the plant salt tolerance response level using the plant salt tolerance response prediction model.

Images