A multi-modal intelligent irrigation decision-making method based on causal inference

Abstract

The application provides a multi-modal intelligent irrigation decision-making method based on causal inference, and knowledge distillation is performed on a pre-trained image feature extractor based on the causal inference, so that the features of crop time sequence information and image information are fully extracted; heterogeneous data is deeply fused by considering multi-modal data. After the knowledge distillation of the image feature extractor, the reasoning speed of the model is faster, the prediction accuracy is higher, the end-to-end water shortage state prediction can be performed based on greenhouse environment data, root system environment data and crop images, the prediction result can be used for guiding crop irrigation, correcting the irrigation frequency and irrigation amount of crops, and realizing more intelligent and comprehensive irrigation decision-making.

Description

Technical Field

[0001] This invention relates to the field of agricultural greenhouse crop irrigation prediction and decision-making technology, specifically to a multimodal intelligent irrigation decision-making algorithm for agricultural greenhouse substrate-cultivated crops. Background Technology

[0002] Smart irrigation integrates information, communication, and control technologies during the irrigation process to accurately predict and control the water requirements and timing of crop growth based on crop needs, thereby achieving optimal water resource utilization while minimizing environmental impact. Smart irrigation possesses the potential for water conservation, energy saving, yield improvement, quality enhancement, and efficiency enhancement, and has become a major trend in modern agricultural irrigation technology. As a cutting-edge field of "smart agriculture," modern large-scale greenhouses, by integrating microclimate monitoring, soil moisture monitoring, and high-definition cameras, are capable of accurately classifying and predicting crop water shortages and managing water usage. Greenhouse substrate-grown crops are highly sensitive to water requirements. Over-irrigation can lead to root hypoxia, root diseases, and nutrient loss from the root system, while under-irrigation can cause water stress, reducing crop quality and slowing growth. Accurately assessing crop water shortage conditions can ensure high-quality and high-yield substrate-grown crops, improve irrigation water utilization efficiency, and achieve energy conservation and increased production.

[0003] In practice, traditional irrigation methods mainly rely on agronomic experience to determine the water shortage status of crops. Some use lysimeters to calculate evapotranspiration and then multiply it by a ratio to determine the irrigation amount. Others trigger quantitative irrigation based on the intensity threshold of outdoor solar radiation or greenhouse light radiation. These methods do not take into account the comprehensive impact of microclimate changes in the greenhouse on crop water consumption, nor do they consider the water stress phenomenon shown by crop morphological information. This makes the planning of irrigation water for greenhouse crops lack a reasonable basis. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal intelligent irrigation decision-making method based on causal inference that combines the characteristics of crop growth environment and image representation, enabling more comprehensive and accurate classification and prediction of crop water shortage status.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal intelligent irrigation decision-making method based on causal inference for classifying and predicting crop water shortage status includes the following steps: Step S1: Collect crop environment and crop image data respectively; Step S2: Perform image enhancement and data preprocessing on the collected data; Step S3: Perform knowledge distillation on the image feature extractor based on causal inference, and obtain a lightweight model of ViT through training and use it for image feature extraction; Step S4: Extract image data features and time-series data features using an image encoder and a time-series encoder; Step S5: Train the multimodal fusion model to obtain multimodal fusion features. Input the multimodal fusion features into the classification head to classify the water shortage state and compare them with the true labels. Calculate the cross-entropy loss and update the model parameters through backpropagation until the classification model converges. Step S6: Apply the trained multimodal fusion model to classify and predict crop water shortage status, thereby achieving classification and prediction of crop water shortage status.

[0006] Preferably, in step S1, the crop environment and crop image data are collected separately, including the following steps: taking a crop image vertically from above the crop every hour using a high-definition camera, and collecting crop environment information every 5 minutes using a microclimate sensor and a soil moisture sensor. The crop environment information includes air temperature, air humidity, water vapor pressure difference, light intensity / photosynthetically active radiation, CO2 concentration, soil / matrix temperature, soil / matrix moisture, soil / matrix EC value, and soil / matrix pH value.

[0007] Preferably, in step 2, the data preprocessing includes data normalization and data imputation for missing data, including the following steps: after camera calibration, image enhancement and image compression, the crop image data is input into ViT; the timestamps in the time series data are converted into time features, a small amount of missing data is imputed, the data is normalized, and the time series data is divided into time series of equal length, aligned with the time of image acquisition, and input into Informer.

[0008] Preferably, the image feature extractor in step 3 uses a pre-trained ViT model, and the model is calibrated and lightweighted based on the knowledge distillation algorithm of causal inference. By integrating three loss functions—alignment loss of sample representation, alignment loss of class representation, and intervention distillation loss—a lightweight ViT model is trained and used for image feature extraction.

[0009] Preferably, the pre-trained image feature extractor ViT is distilled using a causal inference-based knowledge distillation algorithm. The specific process is as follows: Step S31: Define the "Teacher Model" and "Student Model": Use the ViT model and parameters pre-trained on the ImageNet dataset as the teacher model. Use n layers (n≤6) of ViT and the initialized parameters as the student model. ; Step S32: Calculate the alignment loss of the sample representation: for a specific input image sample Teacher Model Output The length of the sample representation logits, Student model Output The length of the sample is represented by logits, which is expressed by the following formula:

[0010]

[0011] in, The sample representation of the student model is transposed. The sample representations of the teacher model are used for alignment loss. The following formula can be used to calculate it:

[0012] in, It is a projection matrix used to map the student model's representation to the same space as the teacher model's representation. It is a weighting coefficient used to control the proportion of loss in the total loss function. The normalized mean square error is represented as follows:

[0013] in, and Each refers to a vector of equal length. Represents the L2 norm; Step S33: Calculate the alignment loss of class representations: Minimize the difference between the class centers of the student model and the teacher model by using the alignment loss of class representations, which is specifically expressed as follows:

[0014] in, It is the total number of classes. These are the weighting coefficients representing the proportion of alignment loss in the total loss function for the control class. Class representation can be expressed by the following formula:

[0015] in, Is class medium sample The class representation is calculated using the feature vectors of all samples in the class and can be considered as the center of the class. Step S34: Calculate the context representation: the sample representation for the teacher model. Sample representation of the student model It can extract context coefficients for different categories. The calculation formula is as follows:

[0016]

[0017] in, Represents the Softmax function. It is a temperature coefficient used to soften the sample representation of the model; then, a context representation is generated based on the class centers. Specifically, it is expressed as follows:

[0018] Step S35: Calculate the intervention distillation loss: Calculate the error in the student model's classification of labels under the contextual intervention of the teacher model, and eliminate the influence of these contextual confusion factors on the prediction results; this is the intervention distillation loss. It can be expressed by the following formula:

[0019] in, It is the first The proportion of each class in the sample It is a true category tag one-hat encoding. It is cross-entropy loss. The KL divergence can be expressed by the following formulas:

[0020]

[0021] in, and Generally refers to a probability distribution with two equal dimensions; Step S36: Backpropagate the loss function: Update the model parameters using the Adam parameter optimizer until the student model converges; the combined loss function can be expressed as follows: .

[0022] Preferably, the pre-trained 12-layer ViT model is distilled into a shallower model (n layers, n≤6) using a causal inference algorithm, and confusion factors are removed by backdoor adjustment in causal inference to ensure that the student model learns the truly useful information from the teacher model.

[0023] Preferably, in step 4, image data features and time-series data features are extracted using an image encoder and a temporal encoder. The temporal data feature extractor is based on probabilistic sparse self-attention, selecting some high-contribution features for feedforward calculation. The following formula can be used to calculate:

[0024] in, Represents the Softmax function. Represents the query matrix. Represents the bond matrix. Representative value matrix, This represents the query matrix after L2 norm normalization. Represents the feature dimension.

[0025] Preferably, in step 4, the image encoder composed of n layers of ViT receives the preprocessed image data and extracts high-dimensional semantic features from the image through a multi-head self-attention layer and a feedforward layer. The self-attention layer generates queries, keys, and values ​​based on image convolutional embeddings. By calculating dot product attention, it can effectively capture complex features and patterns in the image. Meanwhile, the temporal data encoder consists of m layers of Informers. It receives preprocessed time-series data, generates queries, keys, and values ​​based on data embeddings of the time-series data, and selects queries that contribute highly to the results for feedforward computation by calculating probabilistic sparse self-attention, thereby extracting high-dimensional temporal features from the time-series data. The architecture of the Informers is particularly suitable for processing long-series data. Through sparse self-attention mechanism and convolutional operations, it can efficiently extract trend and periodic features from time-series data.

[0026] Preferably, in step 5, the multimodal fusion model achieves deep fusion of image features and temporal features based on cross-attention. The multimodal fusion model is fused and trained through cross-attention, and the specific process is as follows: Multimodal interaction between images and temporal features: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Image features Transform into a query matrix, the length of which is Temporal characteristics The matrix is ​​transformed into a key matrix and a value matrix, as expressed by the following formula:

[0027] in, , , Let represent the transformation matrices of the query matrix, key matrix, and value matrix, respectively. The formula for calculating the cross-attention score using the query matrix, key matrix, and value matrix is ​​as follows:

[0028] Step S52: Classification of crop water shortage status using multimodal features: The multimodal features... A classification head is connected, which consists of two linear layers and a ReLU layer. The linear layers can be represented by the following formula:

[0029] in, and Let represent the weights and biases of the linear layer, respectively. The classification head is used to obtain the sample representation logits, which are specifically expressed as follows:

[0030] Step S53: Model Training: Calculate the cross-entropy loss for the labels using the model's output, perform backpropagation using the loss function, update the model parameters using the Adam parameter optimizer, and train the model to achieve the classification and prediction accuracy for crop water shortage status.

[0031] Compared with the prior art, the present invention has the following beneficial effects: This invention performs knowledge distillation on a pre-trained image feature extractor based on causal inference, which fully extracts features from crop time-series information and image information.

[0032] This invention takes into account multimodal data and deeply fuses heterogeneous data. Through knowledge distillation of the image feature extractor, the model's inference speed is faster and the prediction accuracy is higher. It can perform end-to-end water shortage prediction based on greenhouse environment data, root environment data, and crop images. The prediction results can be used to guide crop irrigation, correct the irrigation frequency and amount, and achieve more intelligent and comprehensive irrigation decisions. Attached Figure Description

[0033] Figure 1 A flowchart illustrating a multimodal intelligent irrigation decision-making method based on causal inference, provided for an embodiment of the present invention; Figure 2 A flowchart of multimodal fusion model training in a multimodal intelligent irrigation decision-making method based on causal inference is provided for embodiments of the present invention; Figure 3 This is a schematic diagram of a multimodal intelligent irrigation decision-making method based on causal inference, provided as an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0035] like Figures 1 to 3 As shown, this invention discloses a multimodal intelligent irrigation decision-making method based on causal inference, used for classifying and predicting crop water shortage states. The specific steps are as follows: Step S1: Collect crop environment and crop image data separately. Use a high-definition camera to take a crop image vertically from above every hour. Collect crop environment information every 5 minutes using a microclimate sensor and a soil moisture sensor. This crop environment information includes air temperature, air humidity, water vapor pressure difference, light intensity / photosynthetically active radiation, CO2 concentration, soil / matrix temperature, soil / matrix moisture, soil / matrix EC value, and soil / matrix pH value.

[0036] Step S2: Perform image enhancement and data preprocessing on the acquired data. After camera calibration, image enhancement, and image compression, the crop image data is compressed to 224*224 pixels for easy input into ViT. The timestamps in the time series data are converted into time features, data is filled in for a small amount of missing data, and the data is normalized. The time series data is then divided into time series of equal length, aligned with the time of image acquisition, and input into Informer.

[0037] Step S3: Knowledge distillation is performed on the image feature extractor based on causal inference. The image feature extractor uses a pre-trained 12-layer ViT model, which is distilled into a shallower model (n layers, n≤6). In this embodiment, the pre-trained 12-layer ViT model is distilled into a 6-layer ViT, and the model is calibrated and lightweighted based on the causal inference knowledge distillation algorithm. Confusion factors are removed using the backdoor adjustment method in causal inference to ensure that the "student model" learns truly useful information from the "teacher model." By integrating three loss functions—sample representation alignment loss, class representation alignment loss, and intervention distillation loss—a lightweight ViT model is trained and used for image feature extraction.

[0038] The knowledge distillation algorithm using causal inference performs model distillation on the pre-trained image feature extractor ViT. The specific process is as follows: Step S31: Define the "teacher model" and "student model": Use the ViT model pre-trained on the ImageNet dataset and its parameters as the teacher model T; use the 6-layer ViT and the initialized parameters as the student model S; Step S32: Calculate the alignment loss of the sample representation: for a specific input image sample Teacher Model Output The length of the sample representation logits, Student model Output The length of the sample is represented by logits, which is expressed by the following formula:

[0039]

[0040] in, The sample representation of the student model is transposed. The sample representations of the teacher model are used for alignment loss. The following formula can be used to calculate it:

[0041] in, It is a projection matrix used to map the student model's representation to the same space as the teacher model's representation. It is a weighting coefficient used to control the proportion of loss in the total loss function. The normalized mean square error is represented as follows:

[0042] in, and Each refers to a vector of equal length. Represents the L2 norm; Step S33: Calculate the alignment loss of class representations: Minimize the difference between the class centers of the student model and the teacher model by using the alignment loss of class representations, which is specifically expressed as follows:

[0043] in, It is the total number of classes. These are the weighting coefficients representing the proportion of alignment loss in the total loss function for the control class. Class representation can be expressed by the following formula:

[0044] in, Is class medium sample The class representation is calculated using the feature vectors of all samples in the class and can be considered as the center of the class. Step S34: Calculate the context representation: the sample representation for the teacher model. Sample representation of the student model It can extract context coefficients for different categories. The calculation formula is as follows:

[0045]

[0046] in, Represents the Softmax function. It is a temperature coefficient used to soften the sample representation of the model; then, a context representation is generated based on the class centers. Specifically, it is expressed as follows:

[0047] Step S35: Calculate the intervention distillation loss: Calculate the error in the student model's classification of labels under the contextual intervention of the teacher model, and eliminate the influence of these contextual confusion factors on the prediction results; this is the intervention distillation loss. It can be expressed by the following formula:

[0048] in, It is the first The proportion of each class in the sample It is a true category tag one-hat encoding. It is cross-entropy loss. The KL divergence can be expressed by the following formulas:

[0049]

[0050] in, and Generally refers to two probability distributions with equal dimensions; Step S36: Backpropagate the loss function: Update the model parameters using the Adam parameter optimizer until the student model converges; the combined loss function can be expressed as follows: .

[0051] Step S4: Extract image data features and temporal data features through an image encoder and a temporal encoder. The image encoder, consisting of 6 ViT layers, receives the preprocessed image data and extracts a 768-dimensional feature vector from the image through a multi-head self-attention layer and a feedforward layer.

[0052] The self-attention layer generates queries, keys, and values ​​through image-based convolutional patch embedding. Calculating dot-product attention effectively captures complex features and patterns in the image. Simultaneously, the temporal data encoder consists of three Informer layers. It receives preprocessed time-series data, generates queries, keys, and values ​​based on data embedding, and calculates probabilistic sparse self-attention. The top 20 queries with the highest attention scores are selected for feedforward computation, extracting 768-dimensional temporal features from the time-series data. The Informer architecture is particularly well-suited for processing long-series data; through sparse self-attention and convolutional operations, it can efficiently extract trend and periodic features from time-series data.

[0053] Image and temporal data features are extracted using an image encoder and a temporal encoder. The temporal data feature extractor is based on probabilistic sparse self-attention, selecting some high-contribution features for feedforward computation. The following formula can be used to calculate:

[0054] in, Represents the Softmax function. Represents the query matrix. Represents the bond matrix. Representative value matrix, This represents the query matrix after L2 norm normalization. Represents the feature dimension.

[0055] Step S5: Train the multimodal fusion model. The multimodal fusion model achieves deep fusion of image features and temporal features based on cross-attention. The multimodal fusion model is fused and trained through cross-attention, and the high-dimensional features extracted by ViT and Informer are input into the cross-attention layer.

[0056] Cross-attention mechanisms can effectively fuse information from different modalities, enabling image features and temporal features to complement and enhance each other in a multimodal space, thereby generating richer and more expressive multimodal fusion feature representations.

[0057] The multimodal fusion feature representation output by the cross-attention mechanism is input to the classification head to classify the water shortage state. After passing through the classifier, logits are generated and compared with the true labels to predict the water shortage state of crops.

[0058] Calculate the cross-entropy loss between the predicted result and the true label, and use the loss function to backpropagate and update the model parameters until the classification model converges. The cross-entropy loss function is shown below:

[0059] in, and This generally refers to two probability distributions with equal dimensions. By optimizing the objective function, prediction errors are gradually reduced, thereby improving the model's prediction accuracy.

[0060] The method for training a multimodal fusion model includes the following sub-steps: Step S51: Perform multimodal interaction between the image and temporal features: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Image features Transform into a query matrix, the length of which is Temporal characteristics The matrix is ​​transformed into a key matrix and a value matrix, as expressed by the following formula:

[0061] in, , , Let represent the transformation matrices of the query matrix, key matrix, and value matrix, respectively. The formula for calculating the cross-attention score using the query matrix, key matrix, and value matrix is ​​as follows:

[0062] Step S52: Classification of crop water shortage status using multimodal features: The multimodal features... A classification head is connected, which consists of two linear layers and a ReLU layer. The linear layers can be represented by the following formula:

[0063] in, and Let represent the weights and biases of the linear layer, respectively. The classification head is used to obtain the sample representation logits, which are specifically expressed as follows:

[0064] Step S53: Model Training: Calculate the cross-entropy loss for the labels using the model's output, perform backpropagation using the loss function, update the model parameters using the Adam parameter optimizer, and train the model to achieve the classification and prediction accuracy for crop water shortage status.

[0065] Step S6: Apply the trained multimodal fusion model to classify and predict crop water shortage status. Use the trained model to classify and predict crop water shortage status based on real-time collected crop environment and image data.

[0066] This invention utilizes a microclimate sensor, a soil moisture sensor, and a high-definition camera to collect crop environment and crop image data. Data augmentation and preprocessing are performed on the collected time-series and image data. Knowledge distillation is applied to the image feature extractor based on causal inference to reduce model complexity and improve the model's applicability and prediction accuracy for this task. Image and time-series data features are extracted using an image encoder and a time-series encoder, respectively. A cross-attention mechanism is used to fuse the extracted high-dimensional features from different modalities, and the multimodal fusion model is trained. The trained model is then applied to classify and predict crop water shortage status. The predicted crop water shortage level can be used for irrigation guidance, adjusting irrigation frequency and amount.

[0067] This invention performs knowledge distillation on a pre-trained image feature extractor based on causal inference, fully extracting features from crop temporal and image information. It also considers multimodal data, deeply fusing heterogeneous data. Through knowledge distillation of the image feature extractor, the model's inference speed is faster and its prediction accuracy is higher. It can perform end-to-end water shortage prediction based on greenhouse environment data, root system environment data, and crop images. The prediction results can be used to guide crop irrigation, correcting irrigation frequency and amount, and achieving more intelligent and comprehensive irrigation decisions.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0069] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal intelligent irrigation decision-making method based on causal inference, used for the classification and prediction of crop water shortage status, characterized in that, Includes the following steps: Step S1: Collect crop environment and crop image data respectively; In step S1, the crop environment and crop image data are collected separately, including the following steps: taking a crop image vertically from above the crop every hour using a high-definition camera, and collecting crop environment information every 5 minutes using a microclimate sensor and a soil moisture sensor. The crop environment information includes air temperature, air humidity, water vapor pressure difference, light intensity / photosynthetically active radiation, CO2 concentration, soil / matrix temperature, soil / matrix moisture, soil / matrix EC value, and soil / matrix pH value. Step S2: Perform image enhancement and data preprocessing on the collected data; In step S2, the data preprocessing includes data normalization and data imputation for missing data, including the following steps: after camera calibration, image enhancement and image compression, the crop image data is input into ViT; after converting the timestamps in the time series data into time features, data imputation for a small number of missing data, and data normalization, the time series data is divided into time series of equal length, aligned with the time of image acquisition, and input into Informer; Step S3: Perform knowledge distillation on the image feature extractor based on causal inference, and obtain a lightweight model of ViT through training and use it for image feature extraction; Step S4: Extract image data features and time-series data features using an image encoder and a time-series encoder; Step S5: Train the multimodal fusion model to obtain multimodal fusion features. Input the multimodal fusion features into the classification head to classify the water shortage state and compare them with the true labels. Calculate the cross-entropy loss and update the model parameters through backpropagation until the classification model converges. Step S6: Apply the trained multimodal fusion model to classify and predict crop water shortage status, thereby achieving classification and prediction of crop water shortage status.

2. The multimodal intelligent irrigation decision-making method based on causal inference according to claim 1, characterized in that, In step S3, the image feature extractor uses a pre-trained ViT model and calibrates and lightweights the model based on the knowledge distillation algorithm of causal inference. By combining three loss functions—alignment loss of sample representation, alignment loss of class representation, and intervention distillation loss—a lightweight ViT model is trained and used for image feature extraction.

3. The multimodal intelligent irrigation decision-making method based on causal inference according to claim 2, characterized in that, The knowledge distillation algorithm using causal inference performs model distillation on the pre-trained image feature extractor ViT. The specific process is as follows: Step S31: Define the "teacher model" and "student model": Use the ViT model and parameters pre-trained on the ImageNet dataset as the teacher model. Use n layers (n≤6) of ViT and the initialized parameters as the student model. ; Step S32: Calculate the alignment loss of the sample representation: for a specific input image sample Teacher Model Output Sample length representation logits, Student model Output The length of the sample is represented by logits, which is expressed by the following formula: in, The sample representation of the student model is transposed. The sample representations of the teacher model are used for alignment loss. The following formula can be used to calculate it: in, It is a projection matrix used to map the representation of the student model to the same space as the representation of the teacher model. It is a weighting coefficient used to control the proportion of loss in the total loss function. The normalized mean square error is represented as follows: in, and Each refers to a vector of equal length. Represents the L2 norm; Step S33: Calculate the alignment loss of class representations: Minimize the difference between the class centers of the student model and the teacher model by using the alignment loss of class representations, which is specifically expressed as follows: in, It is the total number of classes. These are the weighting coefficients representing the proportion of alignment loss in the total loss function for the control class. Class representation can be expressed by the following formula: in, Is class medium sample The class representation is calculated using the feature vectors of all samples in the class and can be considered as the center of the class. Step S34: Calculate the context representation: the sample representation for the teacher model. Sample representation of the student model It can extract context coefficients for different categories. The calculation formula is as follows: in, Represents the Softmax function. It is a temperature coefficient used to soften the sample representation of the model; then, a context representation is generated based on the class centers. Specifically, it is expressed as follows: Step S35: Calculate the intervention distillation loss: Calculate the error in the student model's classification of labels under the contextual intervention of the teacher model, and eliminate the influence of these contextual confusion factors on the prediction results; this is the intervention distillation loss. It can be expressed by the following formula: in, It is the first The proportion of each class in the sample It is a true category tag one-hat encoding. It is cross-entropy loss. The KL divergence can be expressed by the following formulas: in, and Generally refers to a probability distribution with two equal dimensions; Step S36: Backpropagate the loss function: Update the model parameters using the Adam parameter optimizer until the student model converges; the combined loss function can be expressed as follows: 。 4. The multimodal intelligent irrigation decision-making method based on causal inference according to claim 3, characterized in that, The pre-trained 12-layer ViT model is distilled into a shallower model (n layers, n≤6) using a causal inference algorithm. Confusion factors are removed by backdoor adjustment in causal inference to ensure that the student model learns the truly useful information from the teacher model.

5. The multimodal intelligent irrigation decision-making method based on causal inference according to claim 4, characterized in that, In step S4, image data features and time-series data features are extracted using an image encoder and a temporal encoder. The temporal data feature extractor is based on probabilistic sparse self-attention, selecting some high-contribution features for feedforward computation. The following formula can be used to calculate: in, Represents the Softmax function. Represents the query matrix. Represents the bond matrix. Representative value matrix, This represents the query matrix after L2 norm normalization. Represents the feature dimension.

6. The multimodal intelligent irrigation decision-making method based on causal inference according to claim 5, characterized in that, In step S4, the image encoder composed of n layers of ViT receives the preprocessed image data and extracts high-dimensional semantic features from the image through a multi-head self-attention layer and a feedforward layer. The self-attention layer generates query, key, and value based on image convolutional embedding, and can effectively capture complex features and patterns in the image by calculating dot product attention; Meanwhile, the time-series data encoder consists of m layers of Informers. It receives preprocessed time-series data, generates queries, keys, and values ​​based on the data embedding of the time-series data, and selects some queries that contribute highly to the results for feedforward computation by calculating probabilistic sparse self-attention, thereby extracting high-dimensional time-series features from the time-series data. The architecture of the Informers is particularly suitable for processing long-series data. Through sparse self-attention mechanism and convolution operation, it can efficiently extract trend and periodic features from time-series data.

7. The multimodal intelligent irrigation decision-making method based on causal inference according to claim 1, characterized in that, In step S5, the multimodal fusion model achieves deep fusion of image features and temporal features based on cross-attention. The multimodal fusion model is fused and trained through cross-attention, and the specific process is as follows: Multimodal interaction between images and temporal features: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Image features Transform into a query matrix, the length of which is Temporal characteristics The matrix is ​​transformed into a key matrix and a value matrix, as expressed by the following formula: in, , , Let represent the transformation matrices of the query matrix, key matrix, and value matrix, respectively. The formula for calculating the cross-attention score using the query matrix, key matrix, and value matrix is ​​as follows: Step S52: Classification of crop water shortage status using multimodal features: The multimodal features... A classification head is connected, which consists of two linear layers and a ReLU layer. The linear layers can be represented by the following formula: in, and Let represent the weights and biases of the linear layer, respectively. The classification head is used to obtain the sample representation logits, which are specifically expressed as follows: Step S53: Model Training: Calculate the cross-entropy loss for the labels using the model's output, perform backpropagation using the loss function, update the model parameters using the Adam parameter optimizer, and train the model to achieve the classification and prediction accuracy for crop water shortage status.

Images