A method and system for modeling and predicting plant salt tolerance response based on time-series images

Abstract

The present application relates to the technical field of image segmentation, in particular to a plant salt tolerance response modeling and prediction method and system based on time series images. The method comprises pre-processing the obtained plant sample image data; using a plant semantic segmentation model based on U-Net to perform image segmentation on the pre-processed image data; constructing a plant salt tolerance response prediction model based on TimeSformer; and using the plant salt tolerance response prediction model to predict the plant salt tolerance response grade. The plant salt tolerance response prediction modeling method based on time series images significantly improves the efficiency, accuracy and automation level of plant salt tolerance phenotype identification by constructing a prediction process integrating image acquisition, pre-processing, dynamic feature extraction and deep time series modeling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to a plant salt tolerance response modeling and prediction method and system based on time series images. BACKGROUND

[0002] The growth state change of plants under salt stress conditions is a dynamic and continuous physiological response process, which is usually manifested as limited root growth, yellowing of leaves, dryness of leaf edges, dwarfing of plants, and even death. Traditional plant salt tolerance evaluation methods mainly rely on manual periodic observation and sampling to determine physiological indicators (such as SOD activity, chlorophyll content, conductivity, etc.), which have the limitations of long time consumption, large human error, and low processing efficiency, making it difficult to meet the needs of large-scale salt tolerance screening or precise stress response research.

[0003] With the development of image processing and artificial intelligence technology, using computer vision to extract plant phenotype features has become an important direction of plant phenomics. However, most existing researches focus on the analysis of static images, which is difficult to fully capture the continuous response process of plants at different time nodes. In fact, the salt tolerance response of plants has obvious time correlation and process, and only by considering the "dynamic change trajectory" of growth can the salt tolerance ability be evaluated more scientifically and accurately.

[0004] Therefore, it is of great significance to construct a plant salt tolerance response prediction modeling method based on time series images to automatically extract plant stress response trends from "multi-time point image sequences" and build a deep learning model that can predict subsequent salt tolerance states, which can promote intelligent breeding screening, high-throughput stress phenotype identification, and plant stress physiology research. SUMMARY

[0005] In order to solve the above-mentioned problems, the present application provides a plant salt tolerance response modeling and prediction method and system based on time series images.

[0006] In a first aspect, the present application provides a plant salt tolerance response modeling and prediction method based on time series images, which adopts the following technical solution:

[0007] A plant salt tolerance response modeling and prediction method based on time series images, comprising:

[0008] Obtaining plant sample image data;

[0009] Preprocessing the obtained plant sample image data;

[0010] Using a plant semantic segmentation model based on U-Net to perform image segmentation on the preprocessed image data;

[0011] Constructing a plant salt tolerance response prediction model based on TimeSformer;

[0012] Predict the salt tolerance level of plants using a plant salt tolerance response prediction model.

[0013] Furthermore, the preprocessing of the acquired plant sample image data includes image size unification, brightness normalization, and format standardization. Specifically, all images are cropped to rectangular regions containing the complete plant subject and uniformly scaled to a fixed size of 384×384 for batch input into the deep learning model. An adaptive histogram equalization (CLAHE) method is applied to enhance brightness and eliminate color deviations caused by differences in illumination. The standardization formula is:

[0014] ,

[0015] in, These are the original image pixels. It is the overall grayscale mean of the image. It is the standard deviation; at the same time, a high-quality training dataset is constructed, which includes the original plant images and their corresponding segmentation masks. The mask images are binary images, where plant pixels are assigned a value of 1 and the background is assigned a value of 0. Each pair of image-mask samples in the training set is:

[0016] ,

[0017] in, For the i-th input image data, Use its corresponding mask.

[0018] Furthermore, the image segmentation of the preprocessed image data using the U-Net-based plant semantic segmentation model includes extracting spatial hierarchical features of the image layer by layer using the U-Net encoder, transforming the original input image into a feature map with rich semantic information but gradually decreasing resolution. The encoder consists of N repeated convolutional blocks, each block containing two cascaded convolutional layers, a ReLU activation function, and a 2×2 max pooling layer. After each encoding layer, the size of the feature map is halved, and the number of channels is doubled. Let the input image be... , No. The feature map extracted by the layer encoder is The expression for each convolution operation is:

[0019] ,

[0020] in, Indicates the first The convolution kernel of the layer, For the bias term, a conditional label embedding is introduced to encode the experimental meta-information including time point t, salt concentration c and variety type g into a condition vector e, and a channel modulation mechanism is used to dynamically adjust the model feature distribution, assuming that a certain convolutional layer output feature is F, and the modulation method is: ,

[0021] wherein, is the channel scaling and translation factor calculated from the condition vector e by the fully connected layer.

[0022] Further, the image segmentation of the preprocessed image data by the plant semantic segmentation model based on U-Net also includes introducing a time sequence attention mechanism, which enhances the model's perception of time change trends by fusing the semantic features of the current frame and adjacent frames in the encoding stage, wherein the current frame feature is , the previous and next frames are , , the time sequence attention module performs weighted fusion of the previous and next frames of all frames to generate a new frame to fuse the information of the time adjacent frames, improve the temporal consistency of the segmentation result and the sensitivity to stress trends, and the fusion method is represented as:

[0023] ,

[0024] wherein, is a learnable weight parameter generated by the self-attention mechanism, and the specific generation method is: first, perform global pooling operation on the current frame and the previous and next frames to compress the spatial dimension and only retain the channel information, then concatenate the three channel vectors into a matrix , is the vector after global pooling of the corresponding time frame, then perform linear transformation on the matrix to calculate the query, key and value vectors respectively, and then calculate the attention weight matrix.

[0025] Further, the image segmentation of the preprocessed image data by the plant semantic segmentation model based on U-Net also includes gradually restoring the spatial resolution of the feature map by the decoder of U-Net, and restoring the deep semantic information extracted by the encoder to the pixel-level prediction result with the same size as the original image, wherein the input feature map of the decoder is layer is , and the feature of the corresponding encoder is layer is , and the processing process of the decoder is: first, the feature map is enlarged by upsampling the upper layer feature map, represented as:

[0026] ;

[0027] The up-sampled feature map and the shallow layer feature of the encoder are spliced in the channel dimension to fuse local details and global semantic information, and are represented as:

[0028] ,

[0029] Finally, the fused features are refined through two convolution operations to enhance spatial alignment and edge expression, and are represented as:

[0030] ,

[0031] Through the restoration method, the decoder can effectively compensate for the loss of spatial detail information during the encoding stage while recovering the image structure step by step.

[0032] Further, the plant semantic segmentation model based on U-Net is used for image segmentation of the preprocessed image data, which further includes training the U-Net model using a multi-object joint optimization strategy, wherein binary cross entropy (BCE) is used as a basic pixel-level supervision item to measure the accuracy of the model in classifying plant foreground and background, represented as:

[0033] ,

[0034] wherein, is the true value, is the predicted value; and a Dice loss function is introduced to handle plant images with sparse foreground pixels, represented as:

[0035] ,

[0036] wherein is a smoothing term, and a temporal consistency loss term is further introduced to encourage smooth changes in adjacent frame prediction results, represented as:

[0037] ,

[0038] Finally, the overall loss function is a weighted combination of three terms:

[0039] ,

[0040] wherein is a parameter that balances the loss functions.

[0041] Further, the plant salt-tolerant response prediction model based on TimeSformer is constructed, including dividing each frame of image into N image blocks (patches) using TimeSformer, mapping them to high-dimensional space embedding vectors through linear transformation, and forming an initial embedding tensor; then adding spatial position encoding and time position encoding The final embedding vector is represented as:

[0042] ,

[0043] The TimeSformer is composed of multiple stacked temporal Transformer encoding layers, each of which sequentially performs spatial attention and temporal attention, where the input of spatial attention is the patch feature of each frame image , and the query, key and value are obtained by a learnable linear transformation; After spatial attention, the representation of each patch on the time axis is modeled by temporal attention, and after processing all patches and frames, the sequence-level features are extracted from the overall feature sequence by global average pooling:

[0044] ,

[0045] wherein, characterizes the stress response behavior of the whole plant in the entire time sequence, and then the feature is sent to a multi-layer perceptron (MLP) predictor for outputting the final response level: .

[0046] Further, the plant salt tolerance response prediction model is used to predict the plant salt tolerance response level, further comprising adding a channel attention module after the input embedding of the TimeSformer, for enhancing the selective attention ability of the model to the feature channel dimension, wherein based on the SE attention mechanism, a channel attention extraction branch that integrates position information is added, for any feature map after convolution operation, feature compression is performed along the height direction and the width direction respectively, and the position information of the feature in different spatial dimensions is fully retained, represented as:

[0047] ,

[0048] The two feature matrices are then deformed and spliced, and a convolution operation is used to perform feature fusion and position relationship modeling, represented as:

[0049] ,

[0050] wherein, is a weight matrix, is used to reduce the feature dimension and reduce the computational overhead, is a ReLU activation function, and then is separated into and , and another two convolution layers are used to restore the channel number of and , and the final position channel attention is represented as:

[0051] ,

[0052] wherein, is a Sigmod activation function, and is a split operation, and are two weight matrices, the final module attention is composed of SE attention and position attention two channels, all attention information is fed back to the original feature map, and a new feature map added with attention is obtained through convolution operation as output, and the update mechanism of the attention module is specifically represented as:

[0053] ,

[0054] wherein, is the output of the current layer, is a ReLU activation function, is the input of the current layer.

[0055] Further, the plant salt tolerance response prediction model is used to predict the plant salt tolerance response level, further comprising training the plant salt tolerance response prediction model, and during training, each sample input is a plant image sequence segmented by the U-Net , after patch embedding, position coding and channel attention extraction, it is sent to TimeSformer to extract global phenotype response features classify and predict the salt tolerance level of the plant , wherein the loss function is a cross-entropy loss function: ;

[0056] At the same time, in order to maintain the consistency between the output sequences, an inter-frame prediction stability regularization term is introduced:

[0057] ,

[0058] The final total loss function is a weighted combination: .

[0059] In a second aspect, a plant salt tolerance response modeling and prediction system based on time series images comprises:

[0060] A data acquisition module configured to acquire medical images; and acquire plant sample image data;

[0061] A preprocessing module configured to preprocess the acquired plant sample image data;

[0062] The segmentation module is configured to perform image segmentation on the preprocessed image data using a U-Net-based plant semantic segmentation model;

[0063] The training module is configured to build a TimeSformer-based prediction model for plant salt tolerance responses.

[0064] The prediction module is configured to predict the salt tolerance response level of plants using a plant salt tolerance response prediction model.

[0065] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned method for modeling and predicting plant salt tolerance response based on time-series images.

[0066] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide the aforementioned method for modeling and predicting plant salt tolerance response based on time-series images.

[0067] In summary, the present invention has the following beneficial technical effects:

[0068] The proposed method for predicting and modeling plant salt tolerance responses based on time-series images significantly improves the efficiency, accuracy, and automation of plant salt tolerance phenotypic identification by constructing a prediction workflow that integrates image acquisition, preprocessing, dynamic feature extraction, and deep time-series modeling. Compared to traditional manual observation and single-time-point image analysis methods, this method offers the following specific advantages:

[0069] Traditional manual measurement of each plant takes 2–5 minutes, while this method processes a single image in less than 10 seconds, supporting the processing of over 2000 plant images per day, significantly improving screening efficiency. The CNN-LSTM model achieves a 92.3% accuracy rate in predicting plant salt tolerance levels, and the root length prediction regression R² value reaches 0.91, outperforming static models. In tests on three plant species—Arabidopsis thaliana, millet, and alfalfa—the model maintains an accuracy rate of >85%, demonstrating cross-species adaptability. The correlation between image feature prediction values ​​and physiological data such as SOD, Pro, and chlorophyll reaches over 0.83, which can replace some destructive detection methods. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of a plant salt tolerance response modeling and prediction method based on time-series images, according to Embodiment 1 of the present invention.

[0071] Figure 2is a segmentation result comparison chart of a plant salt tolerance response modeling and prediction method based on time sequence images according to Embodiment 1 of the present application. DETAILED DESCRIPTION

[0072] The present application is further described in detail below with reference to the accompanying drawings.

[0073] Embodiment 1

[0074] Reference Figure 1 The plant salt tolerance response modeling and prediction method based on time sequence images according to the present embodiment includes:

[0075] (1) Plant sample processing and image acquisition

[0076] 1) Material preparation and seeding treatment

[0077] Select model or target plant materials such as Arabidopsis thaliana, Setaria viridis, Medicago sativa or Setaria italica, etc. After standard sterilization, seed. Use 70% ethanol for 1 minute, rinse in 1% sodium hypochlorite for 5 minutes, then rinse 3 times with sterile water. Select 1 / 2 MS medium or seedling substrate soil. The seeding density is controlled at where D is the unit area seeding density, N is the number of seeds per culture dish, and A is the effective seeding area of the culture dish.

[0078] 2) Salt stress treatment scheme design

[0079] After the plants germinate, perform stress treatment, design 0, 25, 50, 75, 100, 125, 150 mM NaCl solution concentrations to simulate different stress intensities. Specifically, use root irrigation method to irrigate roots with corresponding concentration NaCl solution every day, at the same time label the stress intensity for subsequent supervised model learning, the specific representation of stress intensity label quantization is:

[0080] ,

[0081] where, is the normalized salt stress level of the ith group of treatments, the NaCl concentration of the ith group, is the control group concentration, is the maximum NaCl concentration. Input the image sequence supervision label into the model for training.

[0082] 3) Time sequence image acquisition design

[0083] To realize the time dynamic modeling of plant salt tolerance response, the program designs a timed image acquisition process to construct a multi-time point image sequence. The specific shooting content includes: top view: observe leaf area, color change; side view: observe plant height, growth potential, leaf curling degree; root system diagram: detect root length and density change. In addition, all image shooting devices are set with fixed angle, uniform light source and white PVC plate background.

[0084] (2) Image preprocessing

[0085] To ensure the consistency of image input between different time points and different samples, size unification, brightness normalization and format standardization processing are needed for the collected images. All images are cropped to a rectangular area containing the complete plant body and scaled to a fixed size of 384x384 for batch input into the deep learning model. In addition, methods such as adaptive histogram equalization (CLAHE) are used for brightness enhancement to eliminate color deviation caused by light differences. The standardization formula is as follows:

[0086] ,

[0087] where, is the pixel of the original image, is the overall gray mean of the image, is the standard deviation.

[0088] In order to realize the subsequent plant semantic segmentation, it is necessary to construct a high-quality training data set, which includes plant original images and their corresponding segmentation masks. The mask image is a binary image, where plant pixels are assigned a value of 1 and background is 0. Each pair of image-mask sample in the training set is:

[0089] ,

[0090] where, is the ith input image data, is its corresponding mask.

[0091] (3) Plant semantic segmentation model based on U-Net

[0092] Semantic segmentation technology is a key basic link to realize plant image time series modeling, which can accurately extract plant regions from complex background and provide high-quality and consistent mask images as input for subsequent feature extraction and time series modeling. By segmenting plant leaves, stems and other structures at the pixel level, semantic segmentation can accurately quantify plant morphology, color and edge changes, especially during the phenotypic degradation process of yellowing, curling and wilting under salt stress conditions, which can improve the stability and accuracy of image phenotype perception. The present application adopts a U-Net plant semantic segmentation network based on a time series attention mechanism, which combines time series attention and conditional embedding mechanism to keep the continuity of the segmentation results in the time dimension. U-Net is a typical encoder-decoder (Encoder-Decoder) architecture semantic segmentation neural network, and the specific structure is divided into encoder and decoder:

[0093] 1) Encoder

[0094] The encoder of U-Net is the down-sampling path part of its network structure, which is used to extract spatial hierarchical features of the image layer by layer, and convert the original input image into a feature map with rich semantic information but gradually reduced resolution. The encoder is composed of N repeated convolution blocks, each block contains two convolution layers (3x3 convolution kernel) connected in series, ReLU activation function and a 2x2 max pooling layer (Max Pooling) for reducing spatial size. After each encoding, the size of the feature map is halved and the number of channels is doubled. Let the input image be , the feature map extracted by the layer encoder is , and the expression of each convolution operation is:

[0095] ,

[0096] wherein, represents the convolution kernel of the layer, is the bias term.

[0097] The response mode of plants under different salt concentrations, treatment days and genotypes conditions is significantly different. The standard U-Net cannot explicitly perceive these treatment backgrounds, resulting in similar images being incorrectly segmented under different conditions, especially in multi-material and multi-gradient treatment experiments. Therefore, the present application introduces a conditional label embedding module to encode experimental meta-information including time point t, salt concentration c and variety type g into a conditional vector e and dynamically adjust the feature distribution of the model through a channel modulation mechanism. Let the output feature of a convolution layer be F, and the modulation method is as follows:

[0098] ,

[0099] wherein, are channel scaling and translation factors computed from the conditional vector e by a fully connected layer.

[0100] In plant salt tolerance response analysis, the time sequence of images is of great significance. A single image cannot reflect the subtle changes of plants during the stress process, especially the early yellowing, leaf drooping and other trend phenotypes. The standard U-Net model processes each image independently, ignoring the continuity between adjacent time frames, resulting in unstable segmentation results and obvious up and down frame jumps. Therefore, the present application introduces a time sequence attention module, which enhances the model's perception of time trend by fusing the semantic features of the current frame and adjacent frames in the encoding stage. Let the current frame feature be , the previous and next frames are , , the time sequence attention module performs weighted fusion of the previous and next frames on all frames to generate a new frame to fuse the information of the time adjacent frames, improve the temporal consistency of the segmentation results and the sensitivity to stress trends, and the specific fusion method is:

[0101] ,

[0102] wherein, is a learnable weight parameter generated by the self-attention mechanism, and the specific generation method is: first, perform global pooling operation on the current frame and the previous and next frames, compress the spatial dimension, and only keep the channel information, then concatenate the three channel vectors into a matrix , is the vector after global pooling of the corresponding time frame. Then perform linear transformation on the matrix to calculate the query (Query), key (Key) and value (Value) vectors respectively:

[0103] ,

[0104] wherein, is a learnable weight, is the attention dimension.

[0105] Then calculate the attention weight matrix :

[0106] ,

[0107] Here, the value of the second row represents the attention weight of the current frame t to the jth frame, that is:

[0108] ,

[0109] In the process of encoding the original image, the encoder extracts features from shallow to deep. The bottom layer extracts edge, color and texture information, such as leaf greenness, spots, and withered edges. The middle layer integrates structural information, such as leaf size and arrangement pattern. The deep layer focuses on high-level semantic information, such as whole plant contour and growth posture. These features are transmitted to the decoder through skip connection, which can effectively aggregate features at different levels and improve the discriminative ability of features.

[0110] 2) Decoder

[0111] The decoder of U-Net is the up-sampling path part of its network structure. Its core function is to gradually restore the spatial resolution of the feature map and restore the deep semantic information extracted by the encoder to the pixel-level prediction result with the same size as the original image. For semantic segmentation tasks, the quality of the decoder directly determines the clarity of the segmentation boundary, the restoration of the outline, and the integrity of the target structure. In plant image processing, the ability to restore details such as slender leaves, jagged edges, and dry spots is crucial. The decoder is composed of multiple up-sampling modules. Each module includes an up-sampling operation, a feature concatenation operation with skip connection to the corresponding encoder layer, and two consecutive 3x3 convolution layers and ReLU activation functions. The main purpose of the up-sampling operation is to enlarge the low-resolution, high-semantic feature map to a higher spatial size, making it gradually approach the resolution of the input image. Let the input feature map of the decoder be the feature of the corresponding encoder be and the processing process of the decoder be:

[0112] 1. Up-sampling

[0113] The feature map is enlarged by up-sampling, which is specifically represented as:

[0114]

[0115] 2. Skip connection concatenation

[0116] The up-sampled feature map is concatenated with the shallow feature of the encoder in the channel dimension, fusing local details and global semantic information. The specific representation is:

[0117]

[0118] 3. Convolution restoration

[0119] Two convolution operations are used to extract and fuse features, enhancing spatial alignment and edge expression. The specific representation is: ​​​​

[0120] ,

[0121] This reduction method enables the decoder to effectively compensate for the loss of spatial detail information during the encoding stage while recovering the image structure step by step. For plant images, the performance of the decoder is particularly crucial. For example, when the leaf edges are blurred and the color changes unevenly after salt stress, the skip connection can restore the leaf edge details and enhance the model's detection ability for phenotypes such as yellowing and curling.

[0122] 3) Loss function and training method

[0123] In the modeling of plant salt tolerance response prediction based on time-series images, the training process of the semantic segmentation model not only needs to ensure accurate segmentation of single images, but also must ensure the continuity of the model in the time dimension and its adaptability to different treatment conditions. Therefore, the improved model TAC-U-Net proposed in this patent adopts a multi-objective joint optimization strategy in the design of the loss function and training method, combining traditional pixel-level segmentation accuracy indicators with regular terms that enhance the model's time-series robustness and conditional adaptability, ensuring the overall performance of the model under multi-dimensional targets. First, binary cross-entropy (BCE) is used as the basic pixel-level supervision term to measure the accuracy of the model in classifying plant foreground and background. Its formula is:

[0124] ,

[0125] where, is the true value, is the predicted value.

[0126] Second, to improve the model's overall grasp of the target area, the Dice loss function is introduced to handle plant images with sparse foreground pixels. The Dice loss can be represented as:

[0127] ,

[0128] where is the smoothing term.

[0129] In addition, to ensure the model's time continuity when processing image sequences, a temporal consistency loss term is introduced to encourage smooth changes in the prediction results of adjacent frames, defined as:

[0130] ,

[0131] Finally, the overall loss function is a weighted combination of the three terms:

[0132] ,

[0133] where are parameters that balance the loss functions.

[0134] The final training parameters of the network are to achieve a good balance between accuracy and consistency. In terms of training methods, the method adopts a three-frame collaborative input mechanism, which sends the current frame and adjacent frames into the network together for temporal attention fusion; the model is only supervised on the current frame to maintain training stability. At the same time, each sample image is attached with a conditional label, which is embedded as a vector and then injected into the model through a modulation layer, effectively improving the model's ability to adapt to experimental processing differences. The hyperparameter settings of TAC-U-Net combine the characteristics of plant salt stress image segmentation tasks and the needs of temporal consistency modeling to ensure segmentation accuracy while considering training stability and computational efficiency. The input of the model is the normalized RGB image and the corresponding conditional label embedding. The encoder part uses a 4-layer convolutional downsampling structure, with a convolution kernel size of 3x3, a step size of 1, a padding of 1, and a ReLU activation function. Batch Normalization is added after each layer to speed up convergence; the decoder part uses Transposed Convolution for upsampling and fuses encoder features through Skip Connection. In the temporal attention module, the time window size is set to 3 frames, using a Multi-head Self-Attention mechanism with 4 attention heads and a channel embedding dimension of 64. The conditional label embedding layer has a dimension of 16 and acts on the intermediate feature layer through ConditionalFiLM. The optimizer uses AdamW with an initial learning rate of 1x10⁻ 4 , a weight decay of 1x10⁻ 5 , a batch size of 8, and a maximum number of training epochs of 200.

[0135] 4) Segmentation image label making

[0136] After semantic segmentation of plant images using U-Net, further processing and labeling of the segmentation results are needed to construct a standardized training dataset or annotation file for model prediction output. In this method, the U-Net model outputs a single-channel mask image (Mask) of the same size as the original image, where each pixel value is a probability value between 0 and 1, representing the confidence that the pixel belongs to the plant foreground. First, the probability mask is thresholded to mark regions with pixel values greater than or equal to the threshold as foreground (value 1) and the rest as background (value 0), resulting in a clear binary mask image. Then, the plant region is identified and the outline is extracted through the contour extraction algorithm, and the basic phenotype characteristics such as leaf boundary, plant height, and area can be obtained. At the same time, to support time series modeling of multi-time point images, the system associates the mask image with the timestamp, processing concentration, sample number, and other information of the original image to generate a one-to-one corresponding image label file. The label file uses a standard CSV format file to record, each record contains: image file name, time point, treatment group, leaf area, plant height, root length, and other index values. In addition, to construct a regression-type supervised task label, the plant physiological data and image mask results are combined to generate a grade label or continuous response value, realizing the numerical coupling of image and salt tolerance phenotype. The physiological indicators used include: SOD activity, conductivity, MDA content, and chlorophyll content. The label generation method is:

[0137] ,

[0138] where, is the set weight, is the image feature score of the ith sample, based on the yellow flower area of the plant in the image and the leaf area change rate. represent the normalized SOD activity, conductivity, MDA content, and chlorophyll content scores, respectively. The final generated stress response index is divided into 5 levels with 0.2 as the dividing line, representing the salt tolerance degree. This label is also the classification target of the subsequent image time series prediction model.

[0139] (4) Plant salt tolerance response prediction model based on TimeSformer

[0140] 1) Model structure

[0141] In the plant salt tolerance response prediction modeling task, the image is not only a static morphological information representation, but also a dynamic phenotype performance process with time continuity. Under salt stress, plants will go through gradual changes from normal appearance to mild yellowing, leaf curling, and severe drying. These small changes in physiology and morphology often appear in a continuous and gradual manner in the image time series. Therefore, traditional static models based on single-frame images are difficult to capture the evolution trend and response rhythm of plants in the time dimension, and may overlook the lag effect and weak performance of stress accumulation in a short period of time. Therefore, it is crucial to introduce a structure with strong time series modeling capability and spatial-temporal feature joint perception capability.

[0142] TimeSformer (Time-Space Transformer) is a visual Transformer structure designed for video time series modeling tasks. It combines spatial perception of image local details and global dependence modeling between frame sequences, and is very suitable for plant salt tolerance response analysis tasks of time series images. Compared with traditional time series neural networks such as LSTM and GRU, TimeSformer can capture long-term and short-term phenotype trends simultaneously, does not rely on strict sequence order input, and has strong robustness to uneven changes between frames. In addition, compared with 3D convolution models, TimeSformer can more efficiently model the time dimension and support flexible adjustment of the number of frames, which is the preferred choice in high-throughput image processing scenarios. The input of TimeSformer is a plant image sequence, denoted as:

[0143] ,

[0144] where T is the number of time frames, and each frame image I has been processed by the above plant semantic segmentation model, retaining only the plant area. To facilitate the processing of the Transformer model, TimeSformer first divides each frame image into fixed-size image patches (patches), sets each frame to N patches, flattens each patch into a vector, and maps it to a high-dimensional space embedding vector through linear transformation to form an initial embedding tensor:

[0145] ,

[0146] where D is the feature dimension. To enable the model to have spatial and temporal position perception capabilities, each patch is overlaid with a spatial position encoding and a temporal position encoding , and the final embedding vector is represented as:

[0147] ,

[0148] The main structure of TimeSformer is composed of multiple stacked temporal Transformer encoding layers, each of which sequentially performs two attention modules: a spatial attention module and a temporal attention module, respectively used to process the inter-patch relationship within each frame of image and model the evolution relationship of the same patch in the time dimension. The spatial attention is processed as follows:

[0149] ,

[0150] where the input is the patch feature of each frame of image , and the query Query, key Key, and value Value are obtained through a learnable linear transformation: The spatial attention focuses on extracting the structural relationship between different leaf regions in the image, such as the spatial difference between the dry and healthy regions. Subsequently, the temporal attention is modeled on the representation of each patch in the time axis after the spatial attention. Let the feature of the patch at position n in each frame be:

[0151] ,

[0152] The calculation method of temporal attention is:

[0153] ,

[0154] where The temporal attention module can be understood as focusing on the "phenotypic change process of the same leaf region over time", which can identify its yellowing, wilting, and other development trends. In order to stabilize the training and increase the expression ability, a LayerNorm layer and a feedforward network (FFN) are used after each spatial and temporal attention:

[0155] ,

[0156] Finally, after all the patches and frames are processed, the model extracts the sequence-level features from the overall feature sequence through global average pooling:

[0157] ,

[0158] where represents the stress response behavior of the whole plant in the entire time sequence. Subsequently, this feature is sent to a multi-layer perceptron (MLP) predictor to output the final response level:

[0159] ,

[0160] When applying TimeSformer to the plant salt tolerance response prediction modeling task, although its original structure has excellent time series modeling and spatial perception capabilities, it cannot effectively extract the dependency relationship between different channels when processing plant image sequences due to the sparse distribution of plant phenotypic changes. To solve the above problems, the patent improves TimeSformer by adding a channel attention module after the input embedding of TimeSformer to enhance the model's selective attention ability to feature channel dimensions.

[0161] Attention mechanisms are widely used in various neural networks to improve network performance. Channel attention mechanisms are a form of attention used to establish attention relationships between features. Channel attention methods represented by SE attention mechanisms effectively improve the performance of CNN networks for image processing tasks. The core idea of SE attention is Squeeze-and-Excitation. The compression operation is a convolution operation after which the spatial information of the entire feature map is compressed, making it aggregate into a single channel feature. This compression method can be implemented using global average pooling. For TimeSformer, this channel attention is specifically represented as:

[0162]

[0163] where, and are the time length and patch number of the input feature map, respectively. Through this squeezing operation, the input feature is compressed to a single feature dimension for subsequent attention generation.

[0164] SE attention mechanism models the channel information obtained by the compression operation through the excitation operation to extract the attention weight between channels. The excitation operation first reduces the feature dimension times through a fully connected layer to compress the channel information; then performs feature transformation through a nonlinear activation function, followed by another fully connected layer to restore the feature dimension to the original channel number; finally, the normalized weight coefficient is obtained through the Sigmoid function and expanded to the original input shape, thereby obtaining the channel attention. This channel attention can be represented as:

[0165]

[0166] where, is the Sigmoid function, ​​is a ReLU activation function, and is a weight matrix.

[0167] Although this channel attention mechanism can effectively mine the dependency between features, thus improving the expression ability of the network, it ignores the position information of the features. Position information is of great significance to guide the spatial distribution of attention. Therefore, the present patent improves the SE attention module and designs a channel attention module LSE that can fuse position information. The module adds a channel attention extraction branch that fuses position information based on the SE attention mechanism. The main process of this branch is as follows: for any feature map after convolution operation, feature compression is performed along the height direction and the width direction respectively, so as to fully retain the position information of the features in different spatial dimensions. This compression method can be realized by using global average pooling, which can be specifically expressed as:

[0168] ,

[0169] In order to further capture the spatial position information in the features, the present patent deforms and splices the two feature matrices, and uses convolution operation to perform feature fusion and position relationship modeling. This process can be expressed as:

[0170] ,

[0171] wherein, is a weight matrix, is used to reduce the feature dimension to reduce the calculation overhead, is a ReLU activation function, and then is separated into and , and the channel numbers of and are restored by using two other convolution layers. The final position channel attention is expressed as:

[0172] ,

[0173] wherein, is a Sigmod activation function, and are segmentation operations, and are two weight matrices.

[0174] The final module attention is composed of SE attention and position attention dual channels, and finally all the attention information is fed back to the original feature map to obtain a new feature map with added attention as output through convolution operation. The update mechanism of the attention module is specifically represented as:

[0175] ,

[0176] wherein, is the output of the current layer, is the ReLU activation function, is the input of the current layer.

[0177] 2) Model training

[0178] The training method of TimeSformer needs to fully consider the actual problems such as time sequence continuity, weak phenotypic change and limited sample quantity in plant salt tolerance response prediction modeling, therefore, the strategy of combining frame-level enhancement, time sequence regularization and conditional control is adopted to improve the stability and generalization ability of the model. During training, the input of each sample is the plant image sequence segmented by U-Net , which is sent to TimeSformer after patch embedding, position encoding and channel attention extraction to extract global phenotypic response features classify and predict the salt tolerance level of plants , wherein the loss function is cross-entropy loss function:

[0179] ,

[0180] In addition, in order to maintain the consistency between the output sequences, a frame prediction stability regularization term is introduced:

[0181] ,

[0182] The final total loss function is a weighted combination:

[0183] ,

[0184] The improved TimeSformer hyperparameter design combines the temporal characteristics of plant salt tolerance response prediction and the multi-task supervision requirements, aiming to enhance the ability to capture subtle phenotypic changes while ensuring global spatiotemporal modeling capabilities. The model input is the RGB plant foreground image sequence processed by TAC-U-Net segmentation and normalization. The number of time frames is set to 8 frames (covering the key time points of the complete salt stress process). The video patch division size is 16x16, and each frame contains 196 patches. The position encoding uses a space and time separation method, and a channel attention module is added after each spatial attention and temporal attention module. The channel number is set to 768. The total network depth is 12 layers of Transformer Block, the number of attention heads (Heads) of the multi-head self-attention mechanism is set to 12, the dimension of each head is 64, the hidden layer dimension (MLPhidden size) is 3072, and the Dropout probability is 0.1. The conditional label embedding dimension is set to 32. The optimizer uses AdamW, the initial learning rate is 5x10⁻ 5 , the weight decay is 0.05, the batch size is 4, and the maximum number of training epochs (Epoch) is 100.

[0185] In summary, the method includes five core steps. First, image time series of plants under different salt concentrations and treatment periods are collected through standardized experimental design, and U-Net model is used for semantic segmentation of images to extract plant foreground area to remove background interference. Second, key image features such as leaf area and yellowing index are extracted from the segmentation results to generate image feature scores, and response labels are constructed in combination with physiological indicators. Then, the processed image sequence is input into the improved TimeSformer model to extract dynamic phenotypic change features. Finally, the model is trained under supervision to output the salt tolerance response level of plants, realizing the prediction and quantification of physiological and phenotypic changes of plants under salt stress.

[0186] Table 1 Comparison of segmentation performance between this method and other U-Net networks

[0187] Model name Intersection over Union Dice coefficient F1 score FPS U-Net 0.812 0.875 0.872 32 Swin-Unet 0.865 0.903 0.905 20 TAC-U-Net (the method) 0.865 0.914 0.911 29

[0188] As shown in Table 1, the experiment was carried out based on the self-acquired salt-tolerant response time-series image dataset of dogtooth grass. The dataset contains 4 salt concentration gradients, 5 treatment time points, a total of 1280 plant samples, each containing multiple consecutive images and corresponding physiological indicators. All images were segmented to extract the plant foreground area by U-Net before entering the time-series modeling, and verified by manual annotation to ensure the accuracy of the segmentation label. The performance of various mainstream semantic segmentation methods was evaluated in this patent, and the results showed that the proposed method performed outstandingly in terms of segmentation accuracy and overall performance. In terms of intersection over union, TAC-U-Net and Swin-Unet were tied for the highest, both reaching 0.865, significantly better than U-Net. In terms of Dice coefficient, TAC-U-Net achieved 0.914, an increase of 1.1 percentage points compared to Swin-Unet and an increase of 3.9 percentage points compared to U-Net. In terms of F1 index, TAC-U-Net reached 0.911, higher than other methods, reflecting the balance between high accuracy and recall rate of foreground pixel identification. Although TAC-U-Net is slightly lower than the original U-Net in inference speed at 29 frames per second, it still meets the application requirements of actual high-throughput phenotype analysis while ensuring high segmentation accuracy and time-series consistency. These results show that TAC-U-Net on this dataset balances accuracy and usability, especially in fine boundary segmentation and stable performance under different salt treatment conditions. As shown in Table 2, the proposed method is compared with other methods. Figure 2 As shown in Table 2, the proposed method is compared with other methods. Figure 2 (a) of (a) and (b) of (b) in FIG. 1 represent the original anemone picture, Figure 2 (b) of (a) and (b) in FIG. 1 represent the original anemone picture, Figure 2 (c) and (d) of (c) and (d) in FIG. 1 represent the segmentation results of the original U-Net, Figure 2 (c) and (d) of (c) and (d) in FIG. 1 represent the segmentation results of the original U-Net, Figure 2 (e) and (f) of (e) and (f) in FIG. 1 represent the segmentation results of Swin-Unet, Figure 2 (e) and (f) of (e) and (f) in FIG. 1 represent the segmentation results of Swin-Unet, Figure 2 (g) and (h) of (g) and (h) in FIG. 1 represent the segmentation results of the proposed method, Figure 2 (g) and (h) of (g) and (h) in FIG. 1 represent the segmentation results of the proposed method,

[0189] Table 2 Comparison of prediction methods in this paper with other methods

[0190] Model name Accuracy F1 Weighted-F1 Early accuracy ResNet50+MLP 74.2% 0.702 0.718 68.0% CNN+LSTM 78.5% 0.743 0.761 72.4% 3D-CNN 80.1% 0.755 0.773 74.9% Swin-Transformer 83.2% 0.781 0.792 77.1% TimeSformer 85.4% 0.807 0.816 80.2% Improved TimeSformer (the method) 87.9% 0.829 0.838 83.5%

[0191] As shown in Table 2, based on the above segmented data set for training, the performance evaluation results on the prediction set show that the method is superior to the comparative method in various indicators. Specifically, in terms of accuracy, the method reaches 87.9%, which is 2.5 percentage points higher than the original TimeSformer, and is significantly better than 3D-CNN, Swin-Transformer, and CNN+LSTM methods. In terms of F1 index, the method achieves 0.829, which is 2.2% higher than the original TimeSformer, and in Weighted-F1, it also achieves an increase from 0.816 to 0.838, reflecting the balanced prediction ability for each class of samples. In terms of early prediction accuracy, the method reaches 83.5%, which is 3.3 percentage points higher than the original TimeSformer, indicating that it has stronger trend recognition and phenotype change capturing ability when processing early stage images under salt stress. These results fully demonstrate that the improved TimeSformer has obvious advantages in early stress prediction scenarios while ensuring overall prediction accuracy, providing reliable technical support for rapid screening and high-throughput analysis of salt tolerance.

[0192] Embodiment 2

[0193] The embodiment provides a plant salt tolerance response modeling and prediction system based on time series images, comprising:

[0194] A data acquisition module configured to acquire medical images and plant sample image data.

[0195] A preprocessing module configured to preprocess the acquired plant sample image data.

[0196] A segmentation module configured to perform image segmentation on the preprocessed image data using a plant semantic segmentation model based on U-Net.

[0197] A training module configured to construct a plant salt tolerance response prediction model based on TimeSformer.

[0198] A prediction module configured to predict the plant salt tolerance response level using the plant salt tolerance response prediction model.

[0199] A computer-readable storage medium having a plurality of instructions stored therein, the instructions being adapted to be loaded and executed by a processor of a terminal device, and implementing the plant salt tolerance response modeling and prediction method based on time series images.

[0200] A terminal device comprises a processor and a computer readable storage medium, the processor is used to realize instructions; the computer readable storage medium is used to store a plurality of instructions, the instructions are suitable for being loaded by the processor and performing the plant salt tolerance response modeling prediction method based on timing image.

[0201] The above are preferred embodiments of the present application, not limited to the protection scope of the present application, therefore: any equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A method for modeling and predicting plant salt tolerance response based on time series images, characterized in that, The method comprises the following steps: acquiring plant sample image data; preprocessing the acquired plant sample image data; performing image segmentation on the preprocessed image data by using a plant semantic segmentation model based on U-Net; constructing a plant salt tolerance response prediction model based on TimeSformer; predicting the plant salt tolerance response level by using the plant salt tolerance response prediction model; The preprocessing of the acquired plant sample image data comprises size unification, brightness normalization and format standardization of the collected images, wherein all the images are cropped to a rectangular region containing the complete plant body and are uniformly scaled to a fixed size of 384x384 for batch input into the deep learning model; and the adaptive histogram equalization CLAHE method is applied for brightness enhancement to eliminate color deviation caused by light difference, and the standardization formula is: ， wherein, is the original image pixel, is the overall gray mean value of the image, is the standard deviation; meanwhile, a high-quality training data set is constructed, which includes plant original images and their corresponding segmentation masks, the mask image is a binary image, wherein the plant pixel is assigned as 1 and the background is 0, and each pair of image-mask sample in the training set is: wherein, is the i input image data, is its corresponding mask; The image segmentation on the preprocessed image data by using the plant semantic segmentation model based on U-Net comprises extracting spatial hierarchical features of the image layer by layer by using an encoder of U-Net, converting the original input image into a feature map with rich semantic information but gradually reduced resolution, and the encoder is composed of N repeated convolution blocks, each block containing two convolution layers in series, a ReLU activation function and a 2x2 max-pooling layer, and after each layer of encoding, the size of the feature map is halved and the number of channels is doubled, assuming that the input image is , the feature map extracted by the first layer encoder is , and the expression of each convolution operation is: ， wherein, denotes the convolution kernel of the layer, is a bias term, while introducing conditional label embedding to encode the experimental meta-information including time point t, salt concentration c, and variety type g into a conditional vector e, and dynamically adjust the model feature distribution through a channel modulation mechanism. Let the output feature of a certain convolution layer be F, and the modulation mode be: ， wherein, are channel scaling and translation factors computed from the conditional vector e by a fully connected layer; The image segmentation of the preprocessed image data by the plant semantic segmentation model based on the U-Net further comprises introducing a time attention mechanism, and the perception ability of the model to the time change trend is enhanced by fusing the semantic features of the current frame and adjacent frames in the encoding stage, wherein the current frame feature is , the previous and subsequent frames are , the time attention module fuses the previous and subsequent frames by weighting all frames , generates a new frame to fuse the information of the time adjacent frames, improves the time consistency of the segmentation result and the sensitivity to the stress trend, and the fusion mode is represented as: ， wherein, is a learnable weight parameter generated by self-attention mechanism, and the generation manner is as follows: firstly, global pooling operation is performed on the current frame and the front and rear frames to compress the spatial dimension and only keep the channel information, and then the vectors of the three channels are spliced into a matrix: ， is the vector after global pooling of the corresponding time frame, followed by a linear transformation using the matrix to compute query, key and value vectors, followed by computation of the attention weight matrix; The image segmentation on the preprocessed image data by using the plant semantic segmentation model based on U-Net further comprises gradually restoring the spatial resolution of the feature map by using the decoder of U-Net, and restoring the deep semantic information extracted by the encoder to the pixel-level prediction result with the same size as the original image, wherein the input feature map of the decoder is the feature of the encoder is the feature of the encoder is the feature of the encoder is The processing process of the decoder is: first, the feature map is enlarged by upsampling the upper layer feature map, which is represented as: ； Then, the up-sampled feature map and the shallow layer feature of the encoder are spliced in the channel dimension to fuse local details and global semantic information, which is represented as: ， Finally, the fused features are refined through two convolution operations to enhance spatial alignment and edge expression, which is represented as: ， Through the restoration method, the decoder effectively compensates for the spatial detail information lost in the encoding stage while recovering the image structure step by step; The image segmentation on the preprocessed image data by using the plant semantic segmentation model based on U-Net further comprises training the U-Net model by using a multi-objective joint optimization strategy, wherein binary cross entropy BCE is used as a basic pixel-level supervision item to measure the accuracy of the model in classifying plant foreground and background, which is represented as: ， where, is the true value, is the predicted value; while introducing the Dice loss function to deal with the plant image with sparse foreground pixels, which is expressed as: ， wherein is a smoothing term, and further introduces a temporal consistency loss term, which encourages the smooth change of the prediction results of adjacent frames, and is expressed as: ， Finally, the overall loss function is a weighted combination of three terms: ， wherein is a parameter balancing the individual loss functions; The TimeSformer-based plant salt tolerance response prediction model is constructed, including dividing each frame of image into N image blocks (patches) by using TimeSformer, mapping into a high-dimensional space embedding vector through linear transformation, and forming an initial embedding tensor; and then superimposing spatial position coding on each patch with time position coding The final embedding vector is represented as: ， The TimeSformer is composed of multiple stacked temporal Transformer encoding layers, each of which sequentially performs spatial attention and temporal attention, wherein the input of the spatial attention is the patch feature of each frame image The query, key and value are obtained through a learnable linear transformation; after the spatial attention, the representation of each patch on the time axis is modeled through the temporal attention, and after the processing of all patches and frames, the sequence-level features are extracted from the overall feature sequence through global average pooling: ， wherein, characterizing the stress response behavior of the whole plant over the whole time series, subsequently feeding this characterization into a multi-layer perceptron, MLP, predictor for outputting a final response rating: .

2. The method of claim 1, wherein, The prediction of the plant salt tolerance response level by using the plant salt tolerance response prediction model further comprises adding a channel attention module after the input embedding of TimeSformer to enhance the selective attention ability of the model to the feature channel dimension, wherein a new channel attention extraction branch that fuses position information is added based on the SE attention mechanism, and for any feature map after convolution operation, feature compression is performed along the height direction and the width direction respectively to fully retain the position information of the features in different spatial dimensions, which is represented as: ， Then, the two feature matrices are spliced after deformation, and convolution operation is used for feature fusion and position relationship modeling, which is represented as: ， wherein, is a weight matrix, for reducing the feature dimension reduces the computational overhead, is a ReLU activation function, followed by is separated into and , the channel numbers of and are restored by using two more convolutional layers, and the final position channel attention is represented as: ， wherein, is a Sigmod activation function, and is a split operation, and are two weight matrices, the final module attention is composed of SE attention and position attention double channels, all attention information is fed back to the original feature map, and a new feature map added with attention is obtained through convolution operation as output, and the updating mechanism of the attention module is specifically represented as: ， wherein, is the output of the current layer, is a ReLU activation function, is the input of the current layer.

3. The method of claim 2, wherein, The method also comprises training the plant salt tolerance response prediction model, wherein each sample input is a plant image sequence segmented by the U-Net After being extracted through the Patch Embedding, position encoding and channel attention, the plant image sequence is sent to the TimeSformer to extract the global phenotype response features The method also comprises classifying the salt tolerance level of the plant For this task, the loss function is selected as the cross-entropy loss function ； Meanwhile, to maintain the consistency between output sequences, a frame prediction stability regularization term is introduced: ， The final total loss function is a weighted combination of: .

4. A time-series image based plant salt tolerance response modeling and prediction system, performing a time-series image based plant salt tolerance response modeling and prediction method as claimed in claim 1, characterized in that, The method comprises the following steps: The data acquisition module is configured to acquire medical images; acquiring plant sample image data; The preprocessing module is configured to preprocess the acquired plant sample image data; The segmentation module is configured to perform image segmentation on the preprocessed image data by using a plant semantic segmentation model based on U-Net; The training module is configured to construct a plant salt tolerance response prediction model based on TimeSformer; The prediction module is configured to predict the plant salt tolerance response level by using the plant salt tolerance response prediction model.

Images