A method and system for deep fusion detection of multimodal soil nutrients

Abstract

This invention discloses a method and system for deep fusion detection of multimodal soil nutrients, belonging to the field of data processing. The method includes the following steps: real-time reception of multimodal data from a sensor suite deployed in the field. The received multimodal data includes soil reflectance spectral data collected by a near-infrared spectral sensor. Compared with the prior art, the advantages of this invention are: real-time reception of spectral, electrochemical, and image data, extraction of deep features after preprocessing and alignment, and introduction of a condition-aware network. The weights of each modality are dynamically adjusted based on auxiliary information such as soil moisture and organic matter content, and finally, a weighted fusion is performed to output the predicted nutrient value. This fully utilizes the complementarity of multimodal information, overcoming the environmental sensitivity and instability of single technologies while retaining their respective advantages, achieving rapid, accurate, and robust soil nutrient detection in the field.

Description

Technical Field

[0001] This invention belongs to the field of data processing, and in particular relates to a method and system for deep fusion detection of multimodal soil nutrients. Background Technology

[0002] Existing soil nutrient detection technologies are mainly divided into two categories: laboratory methods and rapid field detection. Laboratory methods (such as standard chemical analysis) have high accuracy, but are time-consuming, cumbersome, and costly, making them difficult to meet the needs of real-time agricultural decision-making. In rapid field detection, spectroscopic techniques (such as near-infrared) are easily affected by environmental interference such as soil moisture and particle scattering, and are indirect measurements, with accuracy dependent on calibration models; electrochemical sensors (such as ion-selective electrodes) suffer from signal drift, short lifespan, and require frequent calibration, resulting in poor spatial representativeness; image data (such as hyperspectral cameras) are greatly affected by lighting and background, reflecting only surface information and having an indirect correlation with deep nutrients.

[0003] Most on-site rapid detection technologies rely on a single mode, each with its own limitations. This makes it difficult to balance speed, accuracy, and stability in practical applications, resulting in insufficient reliability of the detection results, which requires improvement. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and system for deep fusion detection of multimodal soil nutrients to address the above-mentioned problems.

[0005] The present invention is implemented as follows: a method for deep fusion detection of multimodal soil nutrients includes the following steps:

[0006] It receives multimodal data in real time from a sensor suite deployed in the field. The received multimodal data includes: soil reflectance spectral data (wavelength range 900~1700 nm, resolution 1 nm) collected by a near-infrared spectral sensor, electrochemical data (such as potential values ​​of nitrogen, phosphorus, and potassium, converted to logarithmic ion activity values) in soil extracts collected by an ion-selective electrode array, and RGB image data of the soil surface (resolution 1024×1024 pixels) acquired by a hyperspectral camera; all data are accompanied by timestamps and spatial coordinates.

[0007] Preprocessing is performed on the data of the three modalities to obtain multimodal sample pairs. The multimodal sample pairs are then input into three parallel feature extraction networks to obtain three feature vectors in the same time and space.

[0008] The current soil auxiliary information is input into the condition-aware network to obtain the final weight coefficients of each mode (spectral data, electrochemical data, image data); the condition-aware network is trained end-to-end and dynamically adjusts the weights of each mode according to the input auxiliary information to achieve optimal fusion under different soil conditions;

[0009] The three feature vectors from the same time and space are weighted and summed based on the final weight coefficients to obtain the fused comprehensive feature vector (128 dimensions). The comprehensive feature vector is then input into the prediction head to obtain the predicted values ​​of soil nutrients (such as nitrogen, phosphorus, and potassium) (unit: mg / kg).

[0010] In one embodiment, the present invention provides a method for multimodal deep fusion detection of soil nutrients. The steps of preprocessing the data from the three modalities to obtain multimodal sample pairs, and inputting the multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors at the same time and in the same space, specifically include:

[0011] Preprocessing was performed on the data from the three modalities: spectral data underwent Savitzky-Golay convolution smoothing (11-point window, polynomial order 2) to remove noise, and Standard Normal Variable Transform (SNV) was used to eliminate the effects of particle scattering. Finally, linear interpolation was performed to a fixed wavelength interval. Electrochemical data underwent temperature compensation correction based on temperature sensor readings, and sliding median filtering was used to remove transient pulse interference. RGB images were cropped (uniformed to 512×512), histogram equalization was performed to enhance contrast, and pure soil areas were segmented to exclude straw or shadow interference. Simultaneously, the three types of data from the same sampling point were associated and aligned using timestamps and spatial coordinates to form multimodal sample pairs.

[0012] Multimodal sample pairs are input into three parallel feature extraction networks to obtain three feature vectors in the same time and space. These three feature vectors are projected onto a common feature space (each with a dimension of 128) through independent linear layers for subsequent fusion. The three parallel feature extraction networks are as follows: the spectral feature extractor is a one-dimensional residual convolutional network (1D-ResNet) containing 6 residual blocks, which maps the preprocessed spectral curve (length 800) to a 128-dimensional feature vector; the electrochemical feature extractor is a three-layer fully connected network (64-128-64), which maps the corrected ion activity value (3-dimensional) to a 64-dimensional feature vector; and the image feature extractor uses a convolutional network MobileNetV2 (pre-trained on ImageNet, fine-tuned for soil images) to extract 256-dimensional features before the last layer of global average pooling.

[0013] In one embodiment, the present invention provides a method for deep fusion detection of multimodal soil nutrients. The method involves inputting auxiliary information of the current soil into a condition-aware network to obtain the final weight coefficients (spectral data, electrochemical data, image data) for each modality. The condition-aware network is trained end-to-end, dynamically adjusting the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions. Specifically, this step includes:

[0014] The current soil auxiliary information is input into the condition perception network. The auxiliary information includes real-time humidity sensor reading h (0~100%), soil organic matter content o retrieved from spectral data (unit g / kg estimated using partial least squares regression model), and soil texture features t extracted from image texture (such as clay content, 3D).

[0015] The auxiliary information is concatenated into a 10-dimensional vector. The input is fed into a situation-aware network, which has two layers. First, it passes through a hidden layer (64 dimensions) to obtain the hidden representation. Then, the output layer yields three unnormalized weight logarithms. ,in The final weight coefficients for each mode are then obtained using the Softmax function. ;in, It is the weight matrix of the hidden layer, with a size of 64×10 (64 rows correspond to hidden layer neurons, and 10 columns correspond to the input dimension), used to linearly map the input vector to a 64-dimensional space; is the bias vector of the hidden layer, with a dimension of 64, which is linearly combined with the weights and then added; ReLU is the activation function, introducing non-linearity; h is the output vector of the hidden layer, with a dimension of 64; The weight matrix for the output layer has a size of 3×64 (3 rows correspond to the logarithms of the weights of the three modalities, and 64 columns correspond to the output of the hidden layer), which maps the hidden representation to the three unnormalized logarithms of the weights. The bias vector for the output layer, with dimension 3; These represent the unnormalized weighted logarithms of the three modes: spectroscopy, electrochemistry, and imaging, respectively. These are the final weighting coefficients for the spectral, electrochemical, and image modes, respectively. ;

[0016] By training the condition-aware network end-to-end, the weights of each modality are dynamically adjusted based on the input auxiliary information to achieve optimal fusion under different soil conditions.

[0017] In one embodiment, the present invention provides a method for deep fusion detection of multimodal soil nutrients, further comprising:

[0018] Determine if the data from the three modalities are abnormal;

[0019] If there is an anomaly in a certain modality, the feature vectors of the other two normal modalities are concatenated to form a joint query vector. The nearest neighbor search (e.g., using Euclidean distance) is performed in a pre-built feature library containing historical multimodal samples. The historical sample that is most similar to the current normal modality is matched from the feature library. Then, the feature data of the corresponding abnormal modality in the historical sample is extracted and replaced with the current abnormal data.

[0020] In one embodiment, the present invention provides a method for deep fusion detection of soil nutrients across multiple modalities, wherein determining whether the data from the three modalities are abnormal specifically includes:

[0021] Obtain feature vectors for three modalities , The feature vectors of the two modalities Input the pre-trained reconstruction network Predict another mode eigenvectors Calculate actual characteristics With predictive features Euclidean distance between ,like or modality If the corresponding sensor self-test flag (such as signal loss, exceeding the measurement range, or other hardware anomaly indicators) is true, then the mode is determined. Anomalies, among which, The 95th percentile threshold is used to reconstruct the error distribution based on historical normal data.

[0022] In one embodiment, the present invention provides a method for deep fusion detection of multimodal soil nutrients, further comprising:

[0023] Each time a new set of multimodal data is received, the final predicted value of soil nutrients is stored. When the corresponding time and space laboratory soil nutrient standard value is received (laboratory soil nutrient testing has high accuracy, but its disadvantages are long time consumption, complicated process and high cost), the online learning process is triggered: the relative error between the predicted value and the standard value is calculated. If it exceeds the preset threshold (such as 5%), the elastic weight consolidation (EWC) method is used to adjust the feature extraction network and the situation awareness network (lightweight fine-tuning).

[0024] In one embodiment, the present invention provides a soil nutrient multimodal deep fusion detection system, the soil nutrient multimodal deep fusion detection system comprising:

[0025] The data acquisition module is used to receive multimodal data from the sensor suite deployed in the field in real time. The received multimodal data includes: soil reflectance spectral data (wavelength range 900~1700 nm, resolution 1 nm) collected by the near-infrared spectral sensor, electrochemical data in the soil extract (such as the potential values ​​of nitrogen, phosphorus, and potassium, which have been converted into logarithmic ion activity values) collected by the ion-selective electrode array, and RGB image data of the soil surface (resolution 1024×1024 pixels) acquired by the hyperspectral camera; all data are accompanied by timestamps and spatial coordinates.

[0026] The data processing module is used to preprocess the data of the three modalities respectively to obtain multimodal sample pairs. The multimodal sample pairs are then input into three parallel feature extraction networks to obtain three feature vectors in the same time and space.

[0027] The weight adjustment module is used to input the current soil auxiliary information into the condition perception network to obtain the final weight coefficients of each mode (spectral data, electrochemical data, image data); the condition perception network is trained end-to-end and dynamically adjusts the weights of each mode according to the input auxiliary information to achieve optimal fusion under different soil conditions;

[0028] The prediction acquisition module is used to sum three feature vectors from the same time and space based on the final weight coefficient to obtain a fused comprehensive feature vector (128 dimensions). The comprehensive feature vector is then input into the prediction head to obtain the predicted values ​​of soil nutrients (such as nitrogen, phosphorus, and potassium) (unit: mg / kg).

[0029] In one embodiment, the present invention provides a soil nutrient multimodal deep fusion detection system, the data processing module comprising:

[0030] The data preprocessing unit performs preprocessing on the data from the three modalities separately: spectral data is smoothed using Savitzky-Golay convolution (11-point window, polynomial order 2) to remove noise, and Standard Normal Variable Transform (SNV) is used to eliminate the influence of particle scattering, and finally linearly interpolated to a fixed wavelength interval; electrochemical data is temperature compensated and corrected based on temperature sensor readings, and sliding median filtering is used to remove transient pulse interference; RGB images are cropped (uniformed to 512×512), histogram equalized to enhance contrast, and pure soil areas are segmented to exclude straw or shadow interference; at the same time, the three types of data from the same sampling point are associated and aligned using timestamps and spatial coordinates to form multimodal sample pairs;

[0031] The feature extraction unit is used to input multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space. The three feature vectors are projected onto a common feature space (each with a dimension of 128) through independent linear layers for subsequent fusion. The three parallel feature extraction networks are: a spectral feature extractor is a one-dimensional residual convolutional network (1D-ResNet) containing 6 residual blocks, which maps the preprocessed spectral curve (length 800) to a 128-dimensional feature vector; an electrochemical feature extractor is a three-layer fully connected network (64-128-64), which maps the corrected ion activity value (3-dimensional) to a 64-dimensional feature vector; and an image feature extractor uses a convolutional network MobileNetV2 (pre-trained on ImageNet, fine-tuned on soil images) to extract 256-dimensional features before the last layer of global average pooling.

[0032] In one embodiment, the present invention provides a soil nutrient multimodal deep fusion detection system, wherein the weight adjustment module includes:

[0033] The auxiliary information input unit is used to input the current soil auxiliary information into the condition perception network. The auxiliary information includes the real-time humidity sensor reading h (0~100%), the soil organic matter content o retrieved from the spectral data (in g / kg and estimated using a partial least squares regression model), and the soil texture features t extracted from the image texture (such as the proportion of clay particles, 3D).

[0034] The weight calculation unit is used to concatenate the auxiliary information into a 10-dimensional vector. The input is fed into a situation-aware network, which has two layers. First, it passes through a hidden layer (64 dimensions) to obtain the hidden representation. Then, the output layer yields three unnormalized weight logarithms. ,in The final weight coefficients for each mode are then obtained using the Softmax function. ;in, It is the weight matrix of the hidden layer, with a size of 64×10 (64 rows correspond to hidden layer neurons, and 10 columns correspond to the input dimension), used to linearly map the input vector to a 64-dimensional space; is the bias vector of the hidden layer, with a dimension of 64, which is linearly combined with the weights and then added; ReLU is the activation function, introducing non-linearity; h is the output vector of the hidden layer, with a dimension of 64; The weight matrix for the output layer has a size of 3×64 (3 rows correspond to the logarithms of the weights of the three modalities, and 64 columns correspond to the output of the hidden layer), which maps the hidden representation to the three unnormalized logarithms of the weights. The bias vector for the output layer, with dimension 3; These represent the unnormalized weighted logarithms of the three modes: spectroscopy, electrochemistry, and imaging, respectively. These are the final weighting coefficients for the spectral, electrochemical, and image modes, respectively. ;

[0035] The optimal fusion unit is used to train the condition-aware network end-to-end and dynamically adjust the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions.

[0036] In one embodiment, the present invention provides a soil nutrient multimodal deep fusion detection system, further comprising:

[0037] The modal anomaly detection module is used to determine whether the data in the three modalities are abnormal;

[0038] The abnormal modality replacement module is used to concatenate the feature vectors of the other two normal modalities to form a joint query vector if there is an abnormality in the data of a certain modality. It then performs a nearest neighbor search (e.g., using Euclidean distance) in a pre-built feature library containing historical multimodal samples, matches the historical sample that is most similar to the current normal modality feature from the feature library, and then extracts the feature data of the corresponding abnormal modality from the historical sample to replace the current abnormal data.

[0039] In one embodiment, the present invention provides a soil nutrient multimodal deep fusion detection system, wherein in the modal anomaly judgment module:

[0040] Obtain feature vectors for three modalities , The feature vectors of the two modalities Input the pre-trained reconstruction network Predict another mode eigenvectors Calculate actual characteristics With predictive features Euclidean distance between ,like or modality If the corresponding sensor self-test flag (such as signal loss, exceeding the measurement range, or other hardware anomaly indicators) is true, then the mode is determined. Anomalies, among which, The 95th percentile threshold is used to reconstruct the error distribution based on historical normal data.

[0041] In one embodiment, the present invention provides a soil nutrient multimodal deep fusion detection system, further comprising:

[0042] The iterative update module stores the final predicted soil nutrient values ​​each time a new set of multimodal data is received. When the laboratory soil nutrient standard values ​​corresponding to the time and space are received (laboratory soil nutrient testing has high accuracy, but its disadvantages are long time consumption, cumbersome process, and high cost), the online learning process is triggered: the relative error between the predicted value and the standard value is calculated, and if it exceeds the preset threshold (such as 5%), the elastic weight consolidation (EWC) method is used to adjust the feature extraction network and the situation awareness network (lightweight fine-tuning).

[0043] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention receives three types of data in real time: spectral, electrochemical, and image data. After preprocessing and alignment, deep features are extracted respectively, and a condition-aware network is introduced. The weights of each modality are dynamically adjusted according to auxiliary information such as soil moisture and organic matter content. Finally, the weighted fusion is used to output the predicted nutrient value. This invention makes full use of the complementarity of multimodal information, which not only overcomes the environmental sensitivity and instability of single technologies, but also retains the advantages of each technology, realizing rapid, accurate and robust soil nutrient detection in the field. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the first part of a multimodal deep fusion detection method for soil nutrients provided in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the process of obtaining three feature vectors from three modal data provided in an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of the process for determining the weights of each mode based on soil-aided information, provided in an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of the second part of a multimodal deep fusion detection method for soil nutrients provided in an embodiment of the present invention.

[0048] Figure 5 This is a schematic diagram of the third part of a method for deep fusion detection of multimodal soil nutrients provided in an embodiment of the present invention.

[0049] Figure 6 This is a schematic diagram of the first part of a multimodal deep fusion detection system for soil nutrients provided in an embodiment of the present invention.

[0050] Figure 7 This is a schematic diagram of a data processing module provided in an embodiment of the present invention.

[0051] Figure 8 This is a schematic diagram of the weight adjustment module provided in an embodiment of the present invention.

[0052] Figure 9 This is a schematic diagram of the second part of a multimodal deep fusion detection system for soil nutrients provided in an embodiment of the present invention.

[0053] Figure 10 This is a schematic diagram of the third part of a soil nutrient multimodal deep fusion detection system provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] In one embodiment, such as Figure 1 As shown, a method for deep fusion detection of multimodal soil nutrients includes the following steps:

[0056] Step S1: Receive multimodal data in real time from the sensor suite deployed in the field. The received multimodal data includes: soil reflectance spectral data (wavelength range 900~1700 nm, resolution 1 nm) collected by the near-infrared spectral sensor; electrochemical data (such as the potential values ​​of nitrogen, phosphorus, and potassium, converted to logarithmic ion activity values) in the soil leachate collected by the ion-selective electrode array; and RGB image data of the soil surface (resolution 1024×1024 pixels) acquired by the hyperspectral camera. All data are accompanied by timestamps and spatial coordinates.

[0057] Step S2: Perform preprocessing on the data of the three modalities respectively to obtain multimodal sample pairs. Input the multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space.

[0058] Step S3: Input the current soil auxiliary information into the condition-aware network to obtain the final weight coefficients of each mode (spectral data, electrochemical data, image data); the condition-aware network is trained end-to-end and dynamically adjusts the weights of each mode according to the input auxiliary information to achieve optimal fusion under different soil conditions;

[0059] Step S4: The three feature vectors from the same time and space are weighted and summed based on the final weight coefficients to obtain the fused comprehensive feature vector (dimension 128). The comprehensive feature vector is then input into the prediction head to obtain the predicted values ​​of soil nutrients (such as nitrogen, phosphorus, and potassium) (unit: mg / kg).

[0060] The prediction head is a regression module consisting of a two-layer fully connected network (128→64→3). The entire process is automatically computed within the TensorFlow / PyTorch framework (two mainstream open-source deep learning frameworks), with an inference time of less than 100 milliseconds, meeting real-time detection requirements. This high-efficiency processing capability enables the system to support real-time field monitoring needs, providing timely data support for precision fertilization. Each time the system receives a new set of multimodal data, it can quickly generate nutrient prediction results, meeting the real-time decision-making needs of agricultural fields.

[0061] In one embodiment, such as Figure 2As shown, a method for deep fusion detection of soil nutrients using multiple modalities, in step S2, involves preprocessing the data of the three modalities to obtain multimodal sample pairs, and inputting the multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space. Specifically, this step includes:

[0062] Step S21: Preprocessing is performed on the data of the three modalities respectively: the spectral data is smoothed by Savitzky-Golay convolution (11-point window, polynomial order 2) to remove noise, and the standard normal variable transformation (SNV) is used to eliminate the influence of particle scattering. Finally, it is linearly interpolated to a fixed wavelength interval; the electrochemical data is temperature compensated and corrected based on the temperature sensor readings, and the sliding median filter is used to remove transient pulse interference; the RGB images are cropped (uniformed to 512×512), histogram equalization is performed to enhance contrast, and pure soil areas are segmented to exclude straw or shadow interference; at the same time, the three types of data at the same sampling point are associated and aligned using timestamps and spatial coordinates to form multimodal sample pairs;

[0063] Step S22: Input the multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space. The three feature vectors are projected onto a common feature space (all with a dimension of 128) through independent linear layers for subsequent fusion. The three parallel feature extraction networks are: a spectral feature extractor is a one-dimensional residual convolutional network (1D-ResNet) containing 6 residual blocks, which maps the preprocessed spectral curve (length 800) to a 128-dimensional feature vector; an electrochemical feature extractor is a three-layer fully connected network (64-128-64), which maps the corrected ion activity value (3-dimensional) to a 64-dimensional feature vector; and an image feature extractor uses a convolutional network MobileNetV2 (pre-trained on ImageNet, fine-tuned on soil images) to extract 256-dimensional features before the last layer of global average pooling.

[0064] Three parallel feature extraction networks were specifically designed for the characteristics of different modalities: a one-dimensional residual convolutional network solved the gradient vanishing problem in deep networks through residual connections, enabling the extraction of deep features from spectral data; the lightweight convolutional network MobileNetV2 reduced computational complexity while maintaining high representational capability, making it suitable for edge deployment. The feature vectors output by each network were mapped to a common feature space through independent linear layers, creating conditions for subsequent fusion.

[0065] In one embodiment, such as Figure 3As shown, a method for deep fusion detection of multimodal soil nutrients includes step S3, which involves inputting auxiliary information of the current soil into a condition-aware network to obtain the final weight coefficients (spectral data, electrochemical data, and image data) for each modality. The condition-aware network is trained end-to-end and dynamically adjusts the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions. Specifically, this step includes:

[0066] Step S31: Input the current soil auxiliary information into the condition perception network. The auxiliary information includes the real-time humidity sensor reading h (0~100%), the soil organic matter content o retrieved from the spectral data (unit g / kg estimated using a partial least squares regression model), and the soil texture features t extracted from the image texture (such as the proportion of clay particles, 3D).

[0067] Step S32: Concatenate the auxiliary information into a 10-dimensional vector. The input is fed into a situation-aware network, which has two layers. First, it passes through a hidden layer (64 dimensions) to obtain the hidden representation. Then, the output layer yields three unnormalized weight logarithms. ,in The final weight coefficients for each mode are then obtained using the Softmax function. ;in, It is the weight matrix of the hidden layer, with a size of 64×10 (64 rows correspond to hidden layer neurons, and 10 columns correspond to the input dimension), used to linearly map the input vector to a 64-dimensional space; is the bias vector of the hidden layer, with a dimension of 64, which is linearly combined with the weights and then added; ReLU is the activation function, introducing non-linearity; h is the output vector of the hidden layer, with a dimension of 64; The weight matrix for the output layer has a size of 3×64 (3 rows correspond to the logarithms of the weights of the three modalities, and 64 columns correspond to the output of the hidden layer), which maps the hidden representation to the three unnormalized logarithms of the weights. The bias vector for the output layer, with dimension 3; These represent the unnormalized weighted logarithms of the three modes: spectroscopy, electrochemistry, and imaging, respectively. These are the final weighting coefficients for the spectral, electrochemical, and image modes, respectively. ;

[0068] Step S33: Train the condition-aware network end-to-end and dynamically adjust the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions.

[0069] When the humidity sensor reading h ≥ 30%, the spectral data is severely affected by the moisture absorption band. Increase, making The humidity level is higher than other modes to suppress the interference of humidity on the spectrum (the spectrum of high-humidity samples is less correlated with the true nutrient content, while the electrochemical signal is relatively reliable); when the organic matter content (o) is higher than the median of the sample distribution (e.g., 20 g / kg), the soil usually exhibits a darker color and more obvious aggregate structure. These features are manifested as specific textures and color distributions in hyperspectral RGB images. Increase, image modality weights (Soils with high organic matter content often have more obvious image texture features, thus making better use of image information to assist nutrient prediction).

[0070] In one embodiment, such as Figure 4 As shown, a method for deep fusion detection of soil nutrients using multiple modalities, the method further includes:

[0071] Step S5: Determine whether the data of the three modalities are abnormal;

[0072] Step S6: If there is a data anomaly in a certain modality, the feature vectors of the other two normal modalities are concatenated to form a joint query vector. The nearest neighbor search (e.g., using Euclidean distance) is performed in the pre-built feature library containing historical multimodal samples. The historical sample most similar to the current normal modality features is matched from the feature library. Then, the feature data of the corresponding abnormal modality in the historical sample is extracted and replaced with the current abnormal data.

[0073] Historical sample-based repair methods can provide reasonable feature replacements without interrupting detection, ensuring the system's continuous operation capability even when some sensors fail.

[0074] In one embodiment, such as Figure 4 As shown, a method for deep fusion detection of soil nutrients using multiple modalities, step S5, determining whether the data of the three modalities are abnormal specifically includes:

[0075] Obtain feature vectors for three modalities , The feature vectors of the two modalities Input the pre-trained reconstruction network Predict another mode eigenvectors Calculate actual characteristics With predictive features Euclidean distance between ,like or modality If the corresponding sensor self-test flag (such as signal loss, exceeding the measurement range, or other hardware anomaly indicators) is true, then the mode is determined. Anomalies, among which, The 95th percentile threshold is used to reconstruct the error distribution based on historical normal data.

[0076] That is, indicating different ,For example For spec, For elec, The image is an example; you can also use it. For img, For spec, For elec, only Simply indicate the different modes.

[0077] The system constructs an anomaly detection mechanism based on cross-modal reconstruction at the feature level. It uses features from two normal modes to predict features of a third mode. If the Euclidean distance between the predicted and actual values ​​exceeds the 95th percentile threshold of historical data, the mode is determined to be abnormal. This detection method does not rely on additional hardware and can promptly detect problems such as sensor drift and signal loss at the software level.

[0078] By combining the three modalities in pairs for cross-prediction, the system achieves a triple-redundant anomaly detection mechanism. For example, when the spectral sensor experiences a slight drift, electrochemical and image features may still be able to predict near-realistic spectral features through the reconstruction network. If the actual spectral features deviate significantly from the predicted values, an anomaly alarm is triggered. This multimodal mutual verification method greatly improves the reliability of anomaly detection.

[0079] In one embodiment, such as Figure 5 As shown, a method for deep fusion detection of soil nutrients using multiple modalities, the method further includes:

[0080] Step S7: For each new set of multimodal data received, the final predicted soil nutrient values ​​are stored. When the corresponding time and space laboratory soil nutrient standard values ​​are received (laboratory soil nutrient testing has high accuracy, but its disadvantages are long processing time, complicated procedures, and high costs), the online learning process is triggered: the relative error between the predicted value and the standard value is calculated. If it exceeds a preset threshold (e.g., 5%), the Elastic Weight Consolidation (EWC) method is used to adjust the feature extraction network and the situation awareness network (lightweight fine-tuning).

[0081] The feature extraction network and the condition awareness network are fine-tuned using an elastic weight consolidation method to prevent catastrophic forgetting while retaining historical knowledge. This mechanism enables the model to adapt to seasonal changes in soil type and climate and maintain high accuracy over the long term.

[0082] In one embodiment, such as Figure 6As shown, a soil nutrient multimodal deep fusion detection system includes:

[0083] Data acquisition module 1 is used to receive multimodal data from the sensor suite deployed in the field in real time. The received multimodal data includes: soil reflectance spectral data (wavelength range 900~1700 nm, resolution 1 nm) collected by near-infrared spectroscopy sensor, electrochemical data in soil extract (such as potential values ​​of nitrogen, phosphorus, and potassium, converted into logarithmic ion activity values) collected by ion-selective electrode array, and RGB image data of soil surface (resolution 1024×1024 pixels) acquired by hyperspectral camera; all data are accompanied by timestamps and spatial coordinates.

[0084] Data processing module 2 is used to perform preprocessing on the data of the three modalities respectively to obtain multimodal sample pairs. The multimodal sample pairs are then input into three parallel feature extraction networks to obtain three feature vectors in the same time and space.

[0085] The weight adjustment module 3 is used to input the auxiliary information of the current soil into the condition perception network to obtain the final weight coefficients of each mode (spectral data, electrochemical data, image data); the condition perception network is trained end-to-end and dynamically adjusts the weights of each mode according to the input auxiliary information to achieve optimal fusion under different soil conditions;

[0086] The prediction acquisition module 4 is used to sum the three feature vectors at the same time and in the same space based on the final weight coefficient to obtain the fused comprehensive feature vector (dimension 128). The comprehensive feature vector is then input into the prediction head to obtain the predicted value of soil nutrients (such as nitrogen, phosphorus and potassium) (unit mg / kg).

[0087] Based on multimodal fusion prediction of soil nutrients, more diverse auxiliary information (such as meteorological data and historical fertilization records) can be introduced to enhance the decision-making ability of the condition perception network, thereby achieving more accurate nutrient prediction under different soil types and climatic conditions.

[0088] In one embodiment, such as Figure 7 As shown, a multimodal deep fusion detection system for soil nutrients includes a data processing module 2 comprising:

[0089] The data preprocessing unit 21 is used to perform preprocessing on the data of the three modalities respectively: the spectral data is smoothed by Savitzky-Golay convolution (11-point window, polynomial order 2) to remove noise, and the standard normal variable transformation (SNV) is used to eliminate the influence of particle scattering. Finally, it is linearly interpolated to a fixed wavelength interval; the electrochemical data is temperature compensated and corrected based on the temperature sensor readings, and the sliding median filter is used to remove transient pulse interference; the RGB images are cropped (uniformed to 512×512), histogram equalization is performed to enhance contrast, and pure soil areas are segmented to exclude straw or shadow interference; at the same time, the three types of data at the same sampling point are associated and aligned using timestamps and spatial coordinates to form multimodal sample pairs;

[0090] Feature extraction unit 22 is used to input multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space. The three feature vectors are projected to a common feature space (all with a dimension of 128) through independent linear layers for subsequent fusion. The three parallel feature extraction networks are: a spectral feature extractor is a one-dimensional residual convolutional network (1D-ResNet) containing 6 residual blocks, which maps the preprocessed spectral curve (length 800) to a 128-dimensional feature vector; an electrochemical feature extractor is a three-layer fully connected network (64-128-64), which maps the corrected ion activity value (3-dimensional) to a 64-dimensional feature vector; and an image feature extractor uses a convolutional network MobileNetV2 (pre-trained on ImageNet, fine-tuned on soil images) to extract 256-dimensional features before the last layer of global average pooling.

[0091] Spectral data preprocessing employed Savitzky-Golay convolutional smoothing combined with standard normal transformation, effectively removing high-frequency noise and particle scattering interference. Savitzky-Golay smoothing preserves subtle spectral features through local polynomial fitting, while standard normal transformation corrects for baseline shifts and slope variations caused by uneven soil particle size. The combination of these two methods significantly improves the signal-to-noise ratio and consistency of the spectral data.

[0092] In one embodiment, such as Figure 8 As shown, a multimodal deep fusion detection system for soil nutrients includes a weight adjustment module 3 comprising:

[0093] The auxiliary information input unit 31 is used to input the current soil auxiliary information into the condition perception network. The auxiliary information includes the real-time humidity sensor reading h (0~100%), the soil organic matter content o retrieved from the spectral data (in units of g / kg and estimated using a partial least squares regression model), and the soil texture features t extracted from the image texture (such as the proportion of clay particles, 3D).

[0094] Weight calculation unit 32 is used to concatenate the auxiliary information into a 10-dimensional vector. The input is fed into a situation-aware network, which has two layers. First, it passes through a hidden layer (64 dimensions) to obtain the hidden representation. Then, the output layer yields three unnormalized weight logarithms. ,in The final weight coefficients for each mode are then obtained using the Softmax function. ;in, It is the weight matrix of the hidden layer, with a size of 64×10 (64 rows correspond to hidden layer neurons, and 10 columns correspond to the input dimension), used to linearly map the input vector to a 64-dimensional space; is the bias vector of the hidden layer, with a dimension of 64, which is linearly combined with the weights and then added; ReLU is the activation function, introducing non-linearity; h is the output vector of the hidden layer, with a dimension of 64; The weight matrix for the output layer has a size of 3×64 (3 rows correspond to the logarithms of the weights of the three modalities, and 64 columns correspond to the output of the hidden layer), which maps the hidden representation to the three unnormalized logarithms of the weights. The bias vector for the output layer, with dimension 3; These represent the unnormalized weighted logarithms of the three modes: spectroscopy, electrochemistry, and imaging, respectively. These are the final weighting coefficients for the spectral, electrochemical, and image modes, respectively. ;

[0095] The optimal fusion unit 33 is used to train the condition-aware network end-to-end and dynamically adjust the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions.

[0096] The auxiliary information can be further enriched, such as adding multi-dimensional information such as soil pH, electrical conductivity, and temperature gradient; in addition, the condition-aware network can be optimized (such as introducing multi-task learning to predict weights and nutrient values ​​at the same time, or using reinforcement learning to achieve online adaptive adjustment of weights), so as to achieve more refined modal trade-offs in complex field environments.

[0097] In one embodiment, such as Figure 9 As shown, a multimodal deep fusion detection system for soil nutrients also includes:

[0098] Modal anomaly detection module 5 is used to determine whether the data of the three modalities are abnormal;

[0099] The abnormal modality replacement module 6 is used to concatenate the feature vectors of the other two normal modalities to form a joint query vector if there is an abnormality in the data of a certain modality. It performs a nearest neighbor search (e.g., using Euclidean distance) in a pre-built feature library containing historical multimodal samples, matches the historical sample that is most similar to the current normal modality feature from the feature library, and then extracts the feature data of the corresponding abnormal modality in the historical sample to replace the current abnormal data.

[0100] As a storage medium for historical multimodal samples, the update strategy of the feature library directly affects the accuracy of anomaly remediation. A time-decay-based sample elimination mechanism assigns a time weight to each sample. As time progresses, the weight of older samples gradually decreases. When new samples are added and the library capacity reaches its limit, the system prioritizes eliminating the sample with the lowest weight. This strategy ensures that the feature library reflects recent trends in soil conditions (such as nutrient fluctuations due to seasonal changes), avoids unlimited storage space growth, and retains representative samples, ensuring that historical features most closely resembling the current environment can be matched during anomaly remediation.

[0101] In one embodiment, such as Figure 9 As shown, in a soil nutrient multimodal deep fusion detection system, in modal anomaly judgment module 5:

[0102] Obtain feature vectors for three modalities , The feature vectors of the two modalities Input the pre-trained reconstruction network Predict another mode eigenvectors Calculate actual characteristics With predictive features Euclidean distance between ,like or modality If the corresponding sensor self-test flag (such as signal loss, exceeding the measurement range, or other hardware anomaly indicators) is true, then the mode is determined. Anomalies, among which, The 95th percentile threshold is used to reconstruct the error distribution based on historical normal data.

[0103] Fixed thresholds are ill-suited to the data distribution drift caused by soil type diversity and seasonal variations. The adaptive threshold adjustment mechanism constructs a joint distribution model of soil type, season, and reconstruction error. Based on the soil texture classification (e.g., sandy soil, loam, clay) and the season of sampling (e.g., rainy season, dry season), the system dynamically calculates anomaly detection thresholds applicable to the current scenario. For example, clay may experience significant spectral data fluctuations during the rainy season due to its high water holding capacity. In this case, the system automatically relaxes the reconstruction error threshold for spectral modes, avoiding misclassifying normal fluctuations as anomalies, thus effectively reducing the false alarm rate while maintaining anomaly detection sensitivity.

[0104] In one embodiment, such as Figure 10 As shown, a multimodal deep fusion detection system for soil nutrients also includes:

[0105] The iterative update module 7 is used to store the final predicted soil nutrient values ​​after receiving a new set of multimodal data. When the laboratory soil nutrient standard values ​​corresponding to the time and space are received (the laboratory soil nutrient detection has high accuracy, but the disadvantages are long time consumption, complicated process and high cost), the online learning process is triggered: the relative error between the predicted value and the standard value is calculated. If it exceeds the preset threshold (such as 5%), the elastic weight consolidation (EWC) method is used to adjust the feature extraction network and the situation awareness network (lightweight fine-tuning).

[0106] The system can also be expanded to include real-time crop growth observation data as an auxiliary verification indicator for prediction accuracy. Specifically, the system can connect to crop growth monitoring equipment deployed in the field (such as multispectral drones, chlorophyll content sensors, or canopy thermometers) to obtain current crop growth indicators such as relative chlorophyll content, plant height, leaf nitrogen content, or biomass. Since there is a physiological correlation between crop nutrient stress and soil nutrient supply capacity (e.g., crop leaves typically turn yellow and chlorophyll values ​​decrease when the soil is nitrogen deficient), if the predicted soil nutrient values ​​appear normal, but crop growth indicators remain consistently below the standard curve corresponding to the variety and growth stage, it indicates a potential bias in the prediction. In this case, the system can trigger a mild warning, marking the data set as a "sample to be verified," and prioritizing its use for fine-tuning the feature extraction network and condition-aware network after obtaining subsequent laboratory standard values. This expansion allows iterative updates to not only rely on scarce laboratory sampling but also utilize high-frequency, low-cost crop growth data for cross-validation, thereby capturing dynamic changes in the soil-crop system more promptly and improving prediction reliability during critical crop growth stages.

[0107] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0108] Those skilled in the art will understand that all or part of the processes in the systems described in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0112] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for multimodal deep fusion detection of soil nutrients, characterized in that, The multimodal deep fusion detection method for soil nutrients includes the following steps: It receives multimodal data in real time from a sensor suite deployed in the field. The received multimodal data includes: soil reflectance spectral data collected by a near-infrared spectral sensor, electrochemical data in soil extract collected by an ion-selective electrode array, and RGB image data of the soil surface acquired by a hyperspectral camera; all data are accompanied by timestamps and spatial coordinates. Preprocessing is performed on the data of the three modalities to obtain multimodal sample pairs. The multimodal sample pairs are then input into three parallel feature extraction networks to obtain three feature vectors in the same time and space. The current soil auxiliary information is input into the condition-aware network to obtain the final weight coefficients of each mode; the condition-aware network is trained end-to-end and dynamically adjusts the weights of each mode according to the input auxiliary information to achieve optimal fusion under different soil conditions. The three feature vectors from the same time and space are weighted and summed based on the final weight coefficients to obtain the fused comprehensive feature vector. The comprehensive feature vector is then input into the prediction head to obtain the predicted value of soil nutrients.

2. The method for multimodal deep fusion detection of soil nutrients according to claim 1, characterized in that, The step of preprocessing the data of the three modalities separately to obtain multimodal sample pairs, and inputting the multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space, specifically includes: Preprocessing was performed on the data from the three modalities: spectral data were smoothed using Savitzky-Golay convolution to remove noise, and standard normal variable transformation was used to eliminate the influence of particle scattering. Finally, linear interpolation was performed to a fixed wavelength interval. Electrochemical data were temperature-compensated based on temperature sensor readings, and sliding median filtering was used to remove transient pulse interference. RGB images were cropped, histogram equalized to enhance contrast, and pure soil areas were segmented to exclude straw or shadow interference. At the same time, the three types of data from the same sampling point were associated and aligned using timestamps and spatial coordinates to form multimodal sample pairs. Multimodal sample pairs are input into three parallel feature extraction networks to obtain three feature vectors in the same time and space. These three feature vectors are projected onto a common feature space through independent linear layers for subsequent fusion. The three parallel feature extraction networks are: a spectral feature extractor, which is a one-dimensional residual convolutional network containing six residual blocks, mapping the preprocessed spectral curves to a 128-dimensional feature vector; an electrochemical feature extractor, which is a three-layer fully connected network, mapping the corrected ion activity values ​​to a 64-dimensional feature vector; and an image feature extractor, which uses the MobileNetV2 convolutional network to extract 256-dimensional features before the last layer of global average pooling.

3. The method for multimodal deep fusion detection of soil nutrients according to claim 1, characterized in that, The auxiliary information of the current soil is input into the condition perception network to obtain the final weight coefficients of each mode; The condition-aware network is trained end-to-end and dynamically adjusts the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions. Specifically, this includes: The current soil auxiliary information is input into the condition-aware network. The auxiliary information includes real-time humidity sensor reading h, soil organic matter content o retrieved from spectral data, and soil texture features t extracted from image texture. The auxiliary information is concatenated into a 10-dimensional vector. The input is fed into a situation-aware network, which has two layers. First, the input passes through a hidden layer to obtain the hidden representation. Then, the output layer yields three unnormalized weight logarithms. ,in The final weight coefficients for each mode are then obtained using the Softmax function. ;in, It is the weight matrix of the hidden layer, with a size of 64×10, used to linearly map the input vector to a 64-dimensional space; is the bias vector of the hidden layer, with a dimension of 64, which is linearly combined with the weights and then added; ReLU is the activation function, introducing non-linearity; h is the output vector of the hidden layer, with a dimension of 64; The weight matrix of the output layer has a size of 3×64, which maps the hidden representation to three unnormalized weight logarithms. The bias vector for the output layer, with dimension 3; These represent the unnormalized weighted logarithms of the three modes: spectroscopy, electrochemistry, and imaging, respectively. These are the final weighting coefficients for the spectral, electrochemical, and image modes, respectively. ; By training the condition-aware network end-to-end, the weights of each modality are dynamically adjusted based on the input auxiliary information to achieve optimal fusion under different soil conditions.

4. The method for multimodal deep fusion detection of soil nutrients according to claim 1 or 3, characterized in that, The multimodal deep fusion detection method for soil nutrients also includes: Determine if the data from the three modalities are abnormal; If there is an anomaly in a certain modality, the feature vectors of the other two normal modalities are concatenated to form a joint query vector. The nearest neighbor search is performed in a pre-built feature library containing historical multimodal samples. The historical sample that is most similar to the current normal modality is matched from the feature library. Then, the feature data of the corresponding abnormal modality in the historical sample is extracted and replaced with the current abnormal data.

5. The method for multimodal deep fusion detection of soil nutrients according to claim 4, characterized in that, The specific steps for determining whether the data of the three modalities are abnormal include: Obtain feature vectors for three modalities , The feature vectors of the two modalities Input the pre-trained reconstruction network Predict another mode eigenvectors Calculate actual characteristics With predictive features Euclidean distance between ,like or modality If the corresponding sensor self-test flag is true, then the mode is determined. Anomalies, among which, The 95th percentile threshold is used to reconstruct the error distribution based on historical normal data.

6. The method for multimodal deep fusion detection of soil nutrients according to claim 1, characterized in that, The multimodal deep fusion detection method for soil nutrients also includes: Each time a new set of multimodal data is received, the final predicted value of soil nutrients is stored. When the laboratory soil nutrient standard value corresponding to the time and space is received, the online learning process is triggered: the relative error between the predicted value and the standard value is calculated. If it exceeds the preset threshold, the feature extraction network and the condition awareness network are adjusted using the elastic weight consolidation method.

7. A multimodal deep fusion detection system for soil nutrients, characterized in that, The soil nutrient multimodal deep fusion detection system includes: The data acquisition module is used to receive multimodal data from the sensor suite deployed in the field in real time. The received multimodal data includes: soil reflectance spectral data collected by the near-infrared spectral sensor, electrochemical data in the soil leachate collected by the ion-selective electrode array, and RGB image data of the soil surface acquired by the hyperspectral camera; all data are accompanied by timestamps and spatial coordinates. The data processing module is used to preprocess the data of the three modalities respectively to obtain multimodal sample pairs. The multimodal sample pairs are then input into three parallel feature extraction networks to obtain three feature vectors in the same time and space. The weight adjustment module is used to input the current soil auxiliary information into the condition perception network to obtain the final weight coefficients of each mode. The condition perception network is trained end-to-end and dynamically adjusts the weights of each mode according to the input auxiliary information to achieve optimal fusion under different soil conditions. The prediction acquisition module is used to sum three feature vectors from the same time and space based on the final weight coefficient to obtain a fused comprehensive feature vector. The comprehensive feature vector is then input into the prediction head to obtain the predicted value of soil nutrients.

8. The soil nutrient multimodal deep fusion detection system according to claim 7, characterized in that, The data processing module includes: The data preprocessing unit performs preprocessing on the data of the three modalities respectively: the spectral data is smoothed by Savitzky-Golay convolution to remove noise, and the influence of particle scattering is eliminated by standard normal variable transformation, and finally linearly interpolated to a fixed wavelength interval; the electrochemical data is temperature compensated and corrected based on temperature sensor readings, and instantaneous pulse interference is removed by sliding median filtering; the RGB images are cropped, histogram equalized to enhance contrast, and pure soil areas are segmented to exclude straw or shadow interference; at the same time, the three types of data from the same sampling point are associated and aligned using timestamps and spatial coordinates to form multimodal sample pairs; The feature extraction unit is used to input multimodal sample pairs into three parallel feature extraction networks to obtain three feature vectors in the same time and space. The three feature vectors are projected onto a common feature space through independent linear layers for subsequent fusion. The three parallel feature extraction networks are: a spectral feature extractor is a one-dimensional residual convolutional network containing 6 residual blocks, which maps the preprocessed spectral curves into 128-dimensional feature vectors; an electrochemical feature extractor is a three-layer fully connected network, which maps the corrected ion activity values ​​into 64-dimensional feature vectors; and an image feature extractor uses the MobileNetV2 convolutional network to extract 256-dimensional features before the last layer of global average pooling.

9. The soil nutrient multimodal deep fusion detection system according to claim 7, characterized in that, The weight adjustment module includes: The auxiliary information input unit is used to input the current soil auxiliary information into the condition perception network. The auxiliary information includes real-time humidity sensor reading h, soil organic matter content o retrieved from spectral data, and soil texture features t extracted from image texture. The weight calculation unit is used to concatenate the auxiliary information into a 10-dimensional vector. The input is fed into a situation-aware network, which has two layers. First, the input passes through a hidden layer to obtain the hidden representation. Then, the output layer yields three unnormalized weight logarithms. ,in The final weight coefficients for each mode are then obtained using the Softmax function. ;in, It is the weight matrix of the hidden layer, with a size of 64×10, used to linearly map the input vector to a 64-dimensional space; is the bias vector of the hidden layer, with a dimension of 64, which is linearly combined with the weights and then added; ReLU is the activation function, introducing non-linearity; h is the output vector of the hidden layer, with a dimension of 64; The weight matrix of the output layer has a size of 3×64, which maps the hidden representation to three unnormalized weight logarithms. The bias vector for the output layer, with dimension 3; These represent the unnormalized weighted logarithms of the three modes: spectroscopy, electrochemistry, and imaging, respectively. These are the final weighting coefficients for the spectral, electrochemical, and image modes, respectively. ; The optimal fusion unit is used to train the condition-aware network end-to-end and dynamically adjust the weights of each modality based on the input auxiliary information to achieve optimal fusion under different soil conditions.

10. The soil nutrient multimodal deep fusion detection system according to claim 7 or 9, characterized in that, Also includes: The modal anomaly detection module is used to determine whether the data in the three modalities are abnormal; The abnormal modality replacement module is used to concatenate the feature vectors of the other two normal modalities to form a joint query vector if there is an abnormality in the data of a certain modality. It then performs a nearest neighbor search in a pre-built feature library containing historical multimodal samples, matches the historical sample that is most similar to the current normal modality feature from the feature library, and then extracts the feature data of the corresponding abnormal modality from the historical sample to replace the current abnormal data.

Images