T-SVAE feature extraction strategy and method for improving measurement precision of soil rapidly available potassium through near infrared spectrum by T-SVAE feature extraction strategy

Abstract

The invention relates to the technical field of intelligent detection, and discloses a T-SVAE feature extraction strategy and a method for improving near infrared spectrum soil rapidly available potassium measurement precision by using the T-SVAE feature extraction strategy, and the method comprises the following steps: step 1, collecting soil surface samples of different plots, obtaining near infrared spectrum data of the soil samples by using a Fourier transform near infrared spectrometer, and calculating the near infrared spectrum data of the soil samples; determining the actual content of quick-acting potassium in the soil sample by adopting a national standard method; the method comprises the following steps: 1, acquiring near infrared spectrum data, 2, preprocessing the acquired near infrared spectrum data, and removing impurity signals caused by instrument fluctuation, environmental interference and sample physical form difference, and 3, constructing a Transform and supervision constraint fused variational self-encoding model (T-SVAE). By constructing a variational self-encoding model fusing Transform and supervision constraint, the problem of feature blindness caused by high-dimensional data difficulty, nonlinear modeling limitation and unsupervised learning in near infrared spectrum data processing of a traditional feature extraction method is effectively solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent detection, in particular to a T-SVAE feature extraction strategy and a method for improving the measurement accuracy of soil available potassium by the T-SVAE feature extraction strategy. BACKGROUND

[0002] Soil available potassium is an important indicator for representing the potassium supply capacity of soil, and its content affects the root absorption efficiency, photosynthesis intensity and final yield of crops, and is an important basis for soil nutrient diagnosis and precise fertilization scheme making in modern agriculture. In the sustainable development of agriculture, fast and accurate acquisition of the content of soil available potassium has practical significance for avoiding resource waste and environmental pollution caused by excessive application of potassium fertilizer and preventing crop yield reduction caused by potassium deficiency.

[0003] The current soil available potassium detection methods mainly include traditional chemical detection method and near-infrared spectroscopy detection method. The traditional chemical detection method is represented by the ammonium acetate extraction-flame photometry method, which needs to go through the steps of soil sample drying, grinding, extraction, filtration and instrument determination to realize content quantification. The measurement result of the method is accurate and has good repeatability, and the method has been used as a reference method for soil available potassium content determination for a long time, but the method has obvious defects: the detection period is long, and it takes 2-3 days from pretreatment to output result of a single sample, which cannot meet the demand of modern agriculture for rapid diagnosis of soil nutrients; dangerous chemicals such as strong acid and strong alkali are used in the operation, which is dangerous to the safety of the operator and may also pollute the environment; the detection cost is high, and the total cost of a single sample involves reagent consumption, instrument maintenance and labor cost, which is difficult to apply to batch detection of large-scale field samples.

[0004] In order to overcome the defects of the traditional chemical detection method, the near-infrared spectroscopy detection technology is gradually applied in the field of soil nutrient detection due to its non-destructive, fast and low-cost characteristics. The principle of the near-infrared spectroscopy technology is to use the molecular vibration frequency and frequency absorption characteristics of different components in soil to different wavelengths of near-infrared light, establish a quantitative relationship between the spectral information and the content of available potassium, and realize rapid detection. The spectral acquisition and analysis time of a single sample can be controlled within 30 seconds, and no destructive treatment is needed for the sample, which greatly shortens the detection period. However, when the near-infrared spectroscopy technology is used for soil available potassium detection, it is affected by the characteristics of soil samples and the properties of the technology itself, and faces three main problems, which restrict the measurement accuracy and the range of practical application:

[0005] First, the problem of high-dimensional data processing. A single soil near-infrared spectrum usually contains 1000-4000 wavelength variables, and in field sampling, the sample number is often less than 100 due to the influence of sampling cost and field difference, forming a high-dimensional small sample situation. Direct use of full-band spectral data modeling will increase the model complexity, reduce the calculation efficiency, and also easily introduce redundant variables, aggravate model overfitting and affect the prediction ability of unknown samples.

[0006] Second, the problem of low signal-to-noise ratio. The near-infrared spectrum signal itself has weak absorption, and the physical form difference of soil samples, detection environment fluctuation and instrument noise will introduce a large amount of irrelevant impurity signals in the spectral data. The intensity of these impurity signals is often close to or stronger than the characteristic signal related to the available potassium content, resulting in the effective spectral information being masked and making it difficult to accurately extract information related to the available potassium content.

[0007] Third, the difficulty of processing nonlinear relationship. Soil available potassium is an ionic nutrient, and there is no direct absorption peak in the near-infrared band. The content prediction needs to be realized through indirect association with soil organic matter, clay minerals, water and other components (such as ion complexation, hydrogen bonding). These association relationships are affected by soil organic matter content, clay mineral type (such as the adsorption capacity of montmorillonite to potassium ions is higher than that of kaolinite), soil pH value and other factors, making the spectral data and available potassium content present a nonlinear relationship. Traditional linear modeling methods cannot accurately describe this relationship, resulting in large prediction deviation of the model.

[0008] To break through the above problems, the industry has carried out a lot of research around the feature extraction link, aiming to obtain information highly related to the available potassium content from high-dimensional, noisy and nonlinear spectral data, reduce the data dimension and suppress noise interference. Existing feature extraction methods mainly include traditional linear feature selection methods and deep learning-based feature extraction methods:

[0009] Traditional linear feature selection methods include successive projections algorithm (SPA), competitive adaptive reweighted sampling (CARS) and uninformative variable elimination (UVE). The successive projection algorithm selects the wavelength variable with the smallest collinearity through vector projection operation to reduce redundant information; the competitive adaptive reweighted sampling is based on the weight coefficient of the partial least squares regression model, iteratively retains the wavelength variable with large contribution to the model and eliminates irrelevant variables; the uninformative variable elimination calculates the stability index of spectral variables and noise variables by constructing a random noise matrix, and determines and eliminates variables with stability lower than the threshold as uninformative variables. These methods are simple to calculate and have strong interpretability, and have been applied in the detection of agricultural product quality and the prediction of some soil nutrients, but have obvious shortcomings when used for near-infrared detection of soil available potassium: first, the design is based on linear assumption, which cannot capture the nonlinear relationship in spectral data due to molecular vibration coupling and interaction between components, and the extraction ability of indirect association features of available potassium is insufficient; second, the spectrum is regarded as a set of independent wavelength points, ignoring the sequence dependence between wavelengths, making it difficult to analyze the nonlinear spectral changes caused by soil particle scattering effect, and easy to miss or misselect key features; third, the noise tolerance is poor, when there is strong noise such as instrument drift and environmental interference in the spectral data, the noise signal is easily misjudged as effective feature, resulting in feature extraction deviation and affecting the model accuracy.

[0010] With the development of deep learning technology, feature extraction methods based on generative models have gradually been applied to near-infrared spectroscopy analysis. Variational autoencoder (VAE) has become one of the research directions due to its ability to reduce high-dimensional data and learn features. Standard VAE maps high-dimensional input data to low-dimensional latent space through an encoder to generate the probability distribution of latent features, and then reconstructs the original data through a decoder. Its advantage is that it can learn the deep structural features of data in an unsupervised learning framework, and the regularized latent space distribution reduces the risk of overfitting through KL divergence. In the processing of soil near-infrared spectroscopy, standard VAE can compress the spectral dimension and reduce redundant variables, and can capture more nonlinear features of data than traditional linear methods. However, there are still some defects when standard VAE is used for soil available potassium detection: first, the encoder usually uses a fully connected network structure, ignoring the sequence properties of spectral data (i.e. the correlation and dependence between wavelengths), which cannot effectively capture the global interaction information between different wavebands, resulting in incomplete extraction of information related to available potassium and weakening of weak absorption characteristics of specific functional groups in the latent space compression process; second, the unsupervised learning objective is disconnected from the downstream prediction task—VAE's optimization goal is only data reconstruction and latent space regularization, without considering the target variable of available potassium content, which leads to the learned latent features focusing more on soil physical form, instrument noise, and other irrelevant factors, resulting in a lack of targeted feature extraction; third, it has poor adaptability to complex soil samples. When the organic matter content in the sample fluctuates greatly and the clay mineral types are diverse, it is difficult to distinguish between effective chemical signals and interference signals, and the discriminant ability of latent features is insufficient.

[0011] To solve the problem of standard VAE, researchers have proposed various improvement schemes, but no effective breakthrough has been achieved. For example, some research proposes a smooth prior temporal VAE (SISVAE), which improves the performance of temporal data anomaly detection by regularizing the continuity of the latent representation in the time dimension. However, this method does not design for the characteristics of spectral wavelength sequences, cannot explicitly focus on the importance of different wavebands, and has insufficient capture ability for local sensitive wavebands when processing high-dimensional near-infrared spectra with more than 500 wavebands, making it difficult to balance global trends and local features. Another research proposes a SACVAEGAN model combining conditional VAEGAN and self-attention mechanism, which injects class labels as conditions at the decoding end and uses self-attention mechanism to learn global spectral features, improving classification task accuracy. However, this method only introduces conditional information at the decoding end, and the encoder does not use target variables such as potassium content, resulting in inconsistent feature optimization goals between the encoder and the decoder, complex model structure, significant performance decline when testing across datasets, and poor generalization ability. Some research proposes a multi-scale temporal VAE (MSTVAE), which replaces the single-scale recurrent neural network module with a multi-scale 1D convolution parallel structure to extract local and global time domain features and reduce reconstruction error. However, this model is still an unsupervised learning framework and cannot focus on key feature channels related to target variables, making it prone to gradient vanishing or explosion problems when processing long sequence spectral data and having poor tolerance for imbalanced soil sample distribution scenarios.

[0012] In summary, the existing technology has the following problems in the feature extraction link of soil potassium near-infrared spectrum detection: first, traditional linear feature selection methods cannot handle the nonlinear correlation and wavelength sequence dependence of spectral data, have poor noise tolerance, and insufficient feature extraction relevance; second, standard VAE and existing improved models either cause features to deviate from the target task due to unsupervised learning, or fail to effectively capture global and local spectral features due to structural design defects, or have weak generalization ability and are difficult to adapt to complex soil samples; third, existing methods do not establish an end-to-end framework for collaborative optimization of feature extraction and target prediction, cannot simultaneously satisfy the dimensionality reduction effect, discriminability, and physical interpretability of features, leading to long-term limitations of soil potassium near-infrared detection precision due to high-dimensional data and nonlinear modeling, making it difficult to meet the demand for rapid and accurate detection of soil nutrients in modern agriculture. Therefore, developing a new feature extraction strategy that can effectively handle high-dimensional, noisy, and nonlinear spectral data and focus on target variable-related features is key to improving the precision of near-infrared spectral soil potassium measurement. SUMMARY

[0013] To address the deficiencies of the prior art, the present application provides a T-SVAE feature extraction strategy and a method for improving the precision of near-infrared spectral soil potassium measurement, solving the problems raised in the background art.

[0014] To achieve the above object, the present application is implemented by the following technical solutions: a T-SVAE feature extraction strategy and a method for improving the measurement accuracy of near-infrared spectrum soil available potassium, comprising the following steps:

[0015] Step 1, collecting soil surface samples of different plots, using a Fourier transform near-infrared spectrometer to obtain near-infrared spectrum data of the soil samples, and using a national standard method to determine the actual content of available potassium in the soil samples;

[0016] Step 2, pre-processing the obtained near-infrared spectrum data to remove impurity signals caused by instrument fluctuations, environmental interference and sample physical form differences;

[0017] Step 3, constructing a variational auto-encoding model (T-SVAE) that fuses Transformer and supervised constraints, which is composed of an encoder, a latent space and a sampling mechanism, a decoder and a target regression output branch;

[0018] Step 4, setting a composite loss function containing three types of loss terms, and jointly optimizing and training the T-SVAE model in an end-to-end manner;

[0019] Step 5, using the trained T-SVAE model encoder to forward propagate the pre-processed near-infrared spectrum data and extract the corresponding latent feature vector of each sample;

[0020] Step 6, inputting the extracted latent feature vector into a pre-set regression model to construct a soil available potassium prediction model, and outputting a soil available potassium content prediction result through the prediction model.

[0021] Preferably, the encoder in step 3 comprises a batch normalization layer, a preliminary feature transformation layer and a plurality of stacked TransformerBlock in sequence; the batch normalization layer normalizes the input near-infrared spectrum data to stabilize the network training process; the preliminary feature transformation layer adopts a fully connected network combined with a LeakyReLU activation function to perform linear transformation and nonlinear mapping on the normalized spectrum data; a multi-head self-attention mechanism and a feedforward fully connected network are set inside each TransformerBlock, the multi-head self-attention mechanism calculates the attention scores between the query vector, the key vector and the value vector to capture the global correlation between different dimensional features and wavelength points in the spectrum sequence in parallel, the feedforward fully connected network performs nonlinear transformation on the attention mechanism output features, and the residual connection and layer normalization inside the TransformerBlock promote gradient flow.

[0022] Preferably, the latent space in step 3 assumes that the posterior probability distribution of the encoder output obeys a Gaussian distribution defined by the mean vector and the logarithmic variance vector of the encoder output, and the latent variable is sampled from the Gaussian distribution by a reparameterization trick, so that the latent variable is derivable with respect to the model parameters.

[0023] Preferably, the decoder in step 3 is designed symmetrically with the encoder structure, and sequentially includes a preliminary feature transformation layer, a plurality of stacked TransformerBlocks, and an output layer; the preliminary feature transformation layer adopts a fully connected network combined with a LeakyReLU activation function to perform nonlinear expansion on the latent variable.

[0024] Preferably, the composite loss function in step 4 includes a reconstruction loss, a KL divergence loss, and a regression loss; the reconstruction loss is used to measure the difference between the original near-infrared spectrum data and the reconstructed spectrum data.

[0025] Preferably, the preprocessing method in step 2 is selected from any one of wavelet transform, multivariate scatter correction, standard normal variable transformation, first derivative processing, and Savitzky-Golay smoothing; the wavelet transform separates the true signal and noise through multi-scale decomposition; the multivariate scatter correction and the standard normal variable transformation eliminate the influence of optical path variation and surface scattering; the first derivative processing eliminates baseline drift and enhances the resolution capability of overlapping absorption peaks; and the Savitzky-Golay smoothing removes high-frequency random noise based on local polynomial least squares fitting.

[0026] Preferably, the model training in step 4 adopts the Kennard-Stone algorithm to divide the training set and the test set, the training set accounts for 80% of the total sample size, and the test set accounts for 20% of the total sample size; the Bayesian optimization method is used to optimize the model hyperparameters, including the number of fully connected network layers in the encoder and the decoder, the number of neurons in each layer, the number of attention heads in the Transformer module, the dimension of the latent variable, the Dropout ratio, the learning rate, and the loss function hyperparameters; the early stopping mechanism and the learning rate adjustment mechanism are enabled during the training process; the early stopping mechanism monitors the performance of the validation set, and the training is terminated if there is no improvement within the preset number of rounds; and the learning rate adjustment mechanism reduces the learning rate by a preset factor when the performance of the validation set stagnates.

[0027] Preferably, the regression model in step 6 is selected from any one of gradient boosting machine, support vector regression, random forest regression, partial least squares regression, and logistic regression; the regression model optimizes its own hyperparameters through grid search and cross-validation method, constructs a prediction relationship using the latent feature vector extracted by the T-SVAE model, outputs the prediction result of the soil available potassium content, and evaluates the prediction performance through the coefficient of determination, the root mean square error, the mean absolute error, the performance bias ratio, and the prediction quartile range performance ratio.

[0028] Preferably, the trained T-SVAE model is combined with a regression model and deployed on an edge computing device; the edge computing device has a storage capacity of not less than 1 GB, supports a MicroPython environment and PyTorch, is compatible with ONNXRuntime and Triton inference framework.

[0029] The application provides a T-SVAE feature extraction strategy and a method for improving the measurement accuracy of near-infrared spectrum soil available potassium.

[0030] 1. The application effectively solves the problems of high-dimensional data dilemma, nonlinear modeling limitations and feature blindness caused by unsupervised learning in traditional feature extraction methods in near-infrared spectrum data processing by constructing a variational auto-encoding model that integrates Transformer and supervision constraints. The Transformer module can simultaneously capture the local band correlation and global cross-band interaction of the spectrum sequence, enhancing the perception of key spectral information. The supervision constraint mechanism introduces the available potassium content guidance in the latent space, making the extracted latent features highly relevant to the target measurement task, avoiding irrelevant noise interference, and significantly improving the relevance, robustness and effectiveness of feature extraction, laying a high-quality feature foundation for subsequent soil available potassium content prediction.

[0031] 2. The application can significantly improve the measurement accuracy of near-infrared spectrum soil available potassium, breaking through the precision bottleneck caused by unbalanced spectrum signal-to-noise ratio and complex interference factors in traditional measurement methods. On the one hand, through optimized spectrum preprocessing and the anti-noise characteristics of the T-SVAE model, impurity signals caused by instrument drift, environmental temperature and humidity fluctuations, and soil particle size differences are effectively suppressed. On the other hand, the T-SVAE model can accurately identify the key spectral regions related to available potassium content, construct an ordered latent feature space, reduce the risk of model overfitting, enable the subsequent regression prediction model to output reliable results stably, and have physical interpretability, providing clear technical support for improving measurement accuracy.

[0032] 3. The T-SVAE model trained in the application has strong generalization ability when combined with a regression model, and can adapt to near-infrared spectrum data of different soil types and different geographical regions without frequent model reconstruction for specific scenarios. At the same time, the model can be deployed on edge computing devices to meet the needs of on-site rapid non-destructive testing, significantly shorten the detection period and reduce the detection cost, avoid the drawbacks of using hazardous chemicals in traditional chemical detection methods, provide a technical paradigm for intelligent agricultural equipment research and development, and help the precise diagnosis and efficient management of soil nutrients in agricultural production. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 It is the overall architecture diagram of the T-SVAE model of the application.

[0034] Figure 2 The modeling flowchart of the present application;

[0035] Figure 3 The specific processing effect diagram of the pretreatment method in the embodiment of the present application on the original soil near-infrared spectrum data in the present study;

[0036] Figure 4 The RMSECV distribution diagram of VAE and T-SVAE under different pretreatment conditions of the present application;

[0037] Figure 5 The important wavelength acquisition frequency diagram of VAE and T-SVAE of the present application;

[0038] Figure 6 The latent space distribution diagram of VAE and T-SVAE of the present application;

[0039] Figure 7 The uncertainty comparison diagram in the embodiment of the present application;

[0040] Figure 8 The kernel density distribution comparison diagram of the predicted value and the true value of the T-SVAE-GBM model of the present application on the content of available potassium. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0042] Please refer to the drawings in the specification of the present application Figure 1 , the present application provides a T-SVAE feature extraction strategy and a method for improving the measurement accuracy of near-infrared spectrum soil available potassium, comprising the following steps: Figure 8 Step 1, collect soil surface samples of different plots, use a Fourier transform near-infrared spectrometer to obtain near-infrared spectrum data of the soil samples, and use a national standard method to determine the actual content of available potassium in the soil samples;

[0043] Step 2, pretreat the obtained near-infrared spectrum data to remove impurity signals caused by instrument fluctuations, environmental interference and sample physical form differences;

[0044] Step 3, use the pretreated near-infrared spectrum data as the input of the T-SVAE feature extraction strategy, and extract the feature information of the near-infrared spectrum data;

[0045] Step 3, construct a variational auto-encoding model (T-SVAE) that combines Transformer and supervised constraints, which consists of an encoder, a latent space and a sampling mechanism, a decoder, and a target regression output branch;

[0046] Step 4, set a composite loss function containing three types of loss terms, and train the T-SVAE model in an end-to-end manner;

[0047] The composite loss function includes reconstruction loss, KL divergence loss, and regression loss; the reconstruction loss is used to measure the difference between the original near-infrared spectrum data and the reconstructed spectrum data.

[0048] Step 5, use the trained T-SVAE model encoder to forward propagate the preprocessed near-infrared spectrum data and extract the latent feature vector corresponding to each sample;

[0049] Step 6, input the extracted latent feature vector into a pre-set regression model to construct a soil available potassium prediction model, and output the soil available potassium content prediction result through the prediction model.

[0050] The trained T-SVAE model and regression model can be deployed on an edge computing device; the edge computing device must have at least 1GB of available storage, support MicroPython environment and PyTorch, and be compatible with ONNXRuntime and Triton inference framework.

[0051] Specifically, as shown in Figure 1 Figure 1 The T-SVAE starts from the input spectrum data, and then goes through the encoder, the latent space, and the decoder to reconstruct the spectrum. At the same time, a target regression output branch is derived from the encoder to predict the target value. The entire model is trained by jointly optimizing the reconstruction loss, KL divergence loss, and regression loss.

[0052] The internal structure of the encoder is as follows:

[0053] Batch Normalization (Batch Normalization): Batch normalization is performed on the input data to stabilize the network training process and accelerate model convergence. The calculation formula is as follows:

[0054] x1 = Batch Norm(x)

[0055] ​Initial feature transformation: The batch-normalized spectral data x1 is linearly transformed by a dense layer, which aims to map the original spectral features to a new feature space. Then, a LeakyReLU activation function is applied to introduce nonlinearity, allowing the model to capture more complex data patterns and thus enhancing the robustness of the model learning. The calculation formula is as follows:

[0056] h1 = LeakyReLU(W1x1 + b1);

[0057] where W1 and b1 are the weight matrix and bias vector of the dense layer, respectively.

[0058] TransformerBlock: Then, the features are sent to the TransformerBlock. Inside each TransformerBlock, there are mainly two key components: multi-head self-attention mechanism and feed-forward network. The multi-head self-attention mechanism is used to capture the complex correlations and dependencies between different positions in the input sequence from multiple perspectives, while the feed-forward network is used to nonlinearly transform the output of the attention mechanism to enhance the expressive power of the model.

[0059] Multi-Head Self-Attention: Multi-Head Self-Attention is the essence of the Transformer architecture, allowing the model to simultaneously focus on and integrate information from multiple different representation subspaces (i.e., multiple "heads") when encoding the spectral sequence. The calculation process is summarized as follows:

[0060] MultiHead(Q, K, V) = Concat(head1,..., head h )W o

[0061] where each independent h ead i performs a scaled dot-product attention calculation:

[0062]

[0063] Here, Q (Queries), K (Keys), and V (Values) represent the query vector, key vector, and value vector, respectively, which are obtained by linearly transforming the same input sequence (here, h1) by independent linear transformation matrices . d k represents the dimension of the key vector. When calculating the attention weights, the dot product of Q and K is divided by for scaling, which aims to prevent the case when d kThe larger dot product results are too large, causing the softmax function to enter a region of very small gradients, affecting the stability of training. Multi-head mechanism calculates multiple such attention heads in parallel, concatenates their outputs, and then applies a linear transformation matrix W 0 to obtain the final output. This design allows the model to capture complex correlation patterns between spectral features from different perspectives and representation levels (e.g., one head may focus on baseline changes, while another head may focus on absorption peaks of specific functional groups).

[0064] The output of attention calculation is usually connected in residual connection with the original input and layer normalization is performed to facilitate gradient flow and stable training. Unlike batch normalization, which normalizes the entire batch of features, layer normalization independently normalizes all features for each sample. Its calculation formula can be expressed as:

[0065]

[0066] h2 = LN(h1 + MultiHead(Q, K, V))

[0067] where E[X] and Var[X] are the mean and variance of the current layer input feature X in its feature dimension. γ and β are learnable scaling parameters and translation parameters. ∈ is a very small constant to ensure that the denominator is not zero, maintaining the stability of numerical calculation.

[0068] Feedforward fully connected network: After the output of the multi-head self-attention mechanism, each TransformerBlock contains a network usually composed of two layers of fully connected layers, which performs nonlinear transformation on the attention-weighted fused features to enhance the model's representation ability. To stabilize the training process and accelerate the convergence of the model, layer normalization and residual connection are also added after the output of the feedforward network, and its calculation formula is as follows:

[0069] FFN(h2) = ReLU(h2W2 + b2)W3 + b3

[0070] h3 = LN(h2 + FFN(h2)) (8)

[0071] where W2, b2, W3, and b3 are the weight matrix and bias vector of the feedforward network.

[0072] 4. Latent variable parameter output: After depth processing by one or more TransformerBlocks, the encoder finally maps the obtained deep feature representation h3 to the mean vector μ (denoted as Z mean ) and the log variance vector logσ 2 (denoted as Z log_var). These two vectors jointly define the probabilistic properties of the latent space. The formula is shown as follows:

[0073] μ = Z mean = W μ h3 + b μ (9)

[0074] logσ 2 = Z log_var = W σ h3 + b σ (10)

[0075] where W μ , b μ , W σ , and b σ are the weights and biases of the last two fully connected layers.

[0076] In the T-SVAE model, the core assumption is that the posterior distribution q φ (Z|X) output by the encoder follows a Gaussian distribution, i.e. ). Here, μ is the mean vector Z mean output by the encoder, and the variance σ 2 is calculated through the logarithmic variance vector, i.e. σ 2 = exp(Z log_var ). In order to enable end-to-end effective training of the entire model through gradient backpropagation, T-SVAE uses a reparameterization trick to sample the latent variable Z from this learned posterior distribution. The specific sampling process is as follows:

[0077]

[0078] In the formula, ⊙ represents element-wise multiplication of vectors, and ∈ is a noise vector randomly sampled from a standard normal distribution. In this way, randomness is carried by external noise ∈, and the latent variable Z is derivable with respect to the model parameters μ and Z log_var . This is crucial for VAE-type model optimization, as it allows the use of gradient-based optimization algorithms to train the entire network.

[0079] Preliminary feature transformation: the latent variable Z is preliminarily nonlinearly expanded through a fully connected network and a LeakyReLU activation function.

[0080] h4 = LeakyReLU(W4Z + b4)

[0081] TransformerBlock: similar to the encoder, one or more TransformerBlock is applied to the feature h4 for inverse deep transformation to restore the complex structure of the spectral sequence.

[0082] h5 = LN(h4 + MultiHead(h4)) (13)

[0083] FFN(h5) = ReLU(W5h5 + b5) * W6 + b 5* 6( 14)

[0084] h6 = LN(h5 + FFN(h5)) (15)

[0085] Reconstruction output: Finally, the decoder-processed feature h6 is mapped back to the dimension of the original spectrum through a fully connected layer, resulting in the reconstructed spectrum X'.

[0086] X' = W7h6 + b7 (16)

[0087] where W4, b4, W5, b 5， 6， 6， W7, b7 are the learnable parameters of each layer in the decoder.

[0088] In this study, to ensure that the regression task can utilize the richest feature information and stabilize the training process, we concatenate the output of the last hidden layer of the encoder with the latent mean vector Z mean and use them as the input of the regression branch. The final output of this regression branch is the predicted value of the target variable y To enhance the generalization ability of the model and prevent overfitting, we introduce a Dropout layer in the regression branch. Its calculation formula is as follows:

[0089] h7 = LeakyReLU(W8Z mean +b8)

[0090] h8 = Dropout(h7, rate = p)

[0091]

[0092] where W8, b8, W9, b9 are the weights and biases of the regression branch, and p is the ratio of Dropout. By introducing and optimizing this regression branch together with the autoencoder part, the model is explicitly guided to learn latent features that not only can effectively reconstruct the input spectrum but also have strong discriminability for the target variable.

[0093] The training objective of the T-SVAE model is to minimize the composite loss function. This loss function is mainly composed of the following three core parts, which together guide the learning process of the model and ensure that the model achieves a balance in feature extraction, data reconstruction, and target prediction.

[0094] L​​​total = λ1L recon + λ2L reg + βL KL

[0095] Reconstruction loss (L recon ): This loss term is used to measure the difference between the original input spectrum X and the reconstructed spectrum X' obtained through the encoding-decoding process. For continuous spectral data, the Mean Squared Error (MSE) is usually adopted as the measurement standard for reconstruction loss, which is calculated as follows:

[0096]

[0097] where N represents the number of samples involved in the calculation. This loss term ensures that the learned latent variable Z retains enough information to recover the original input spectrum as accurately as possible.

[0098] KL divergence loss (L KL ): This loss term plays the role of a regularization term in the VAE framework. It measures the difference or distance between the posterior probability distribution q φ (Z|X) output by the encoder and the preset latent space prior probability distribution P(Z) (usually a standard normal distribution ). Typically, the prior distribution P(Z) is set as a standard normal distribution KL divergence loss encourages the latent space distribution learned by the encoder to be more regular and smooth, helping to avoid "holes" or discontinuous regions in the latent space, thereby improving the model's generalization ability and the quality of generating new samples. Its specific calculation formula is:

[0099]

[0100] where D is the dimension size of the latent variable, μ j and are the mean and variance of the jth dimension of the latent variable, respectively.

[0101] Regression loss (L reg ): This loss term is used to measure the difference between the predicted available potassium content y by the target regression branch and the true available potassium content y. For continuous regression targets, the MSE is also used as the regression loss measurement, with the formula:

[0102]

[0103] In the total loss function, the values of hyperparameters λ1, λ2 and β need to be adjusted and optimized according to the specific task and data. They are used to balance the relative importance between the reconstruction-regularization task of the autoencoder and the supervised regression task.

[0104] When performing quantitative analysis of data, a variety of different evaluation indicators are usually used to comprehensively and accurately evaluate the performance of a model. In this study, indicators widely used in machine learning and chemometrics were mainly selected to comprehensively measure the calibration ability and prediction ability of unknown samples of the model. The evaluation indicators of data quantitative model mainly include the coefficient of determination (R 2 ), root mean square error (RMSE), mean absolute error (MAE), performance bias ratio (RPD) and prediction interquartile range ratio (RPIQ).

[0105] The coefficient of determination, also known as the goodness of fit, measures the degree of explanation of the total variability of the data by the model. Its value range is usually between 0 and 1. R 2 The closer to 1, the higher the correlation between the predicted value and the true value of the model, the stronger the explanation ability of the dependent variable, and the better the fitting effect. The calculation formula is as follows:

[0106]

[0107] In the formula, n represents the number of samples, y i is the true available potassium content of the i-th sample, is the available potassium content predicted by the model for the i-th sample, is the average value of the true available potassium content of all samples. The coefficients of determination of the training set and the test set are denoted as R and R

[0108] The root mean square error is a commonly used indicator to measure the average deviation between the predicted value and the true value of the model, and it is sensitive to outliers in the prediction error. The smaller the value of RMSE, the higher the prediction accuracy of the model. The calculation formula is as follows:

[0109]

[0110] Similarly, the root mean square errors of the training set and the test set are denoted as RMSEC and RMSEP. Ideally, the values of RMSEC and RMSEP are both low and not significantly different. Lower and close RMSEC and RMSEP values usually indicate that the model has achieved a good balance between goodness of fit and generalization ability, and has high stability and reliability.

[0111] The mean absolute error represents the arithmetic mean of the absolute deviation between the predicted value and the true value of the model. Compared with RMSE, MAE is less sensitive to outliers because it does not square the error. The smaller the value of MAE, the higher the prediction accuracy of the model. The calculation formula is as follows:

[0112]

[0113] The mean absolute error of the training set and the test set are denoted as MAEC and MAEP, respectively, which are the core indicators for evaluating the prediction accuracy of the regression model.

[0114] RPD is a key performance indicator widely used in the field of chemometrics, which comprehensively evaluates the relative accuracy and reliability of the model prediction results by comparing the standard deviation of the sample true value (SD) with the standard deviation of the prediction error of the model on the prediction set (RMSEP). The calculation formula is as follows:

[0115]

[0116] In the formula, SD usually refers to the standard deviation of the true value of the test set, and n is the number of samples in the test set. When RPD<1.5, the model prediction ability is poor; when 1.5≤RPD<2.0, the model can only roughly distinguish or screen samples; when 2.0≤RPD<2.5, the model has good prediction ability; when RPD≥2.5, the model has excellent prediction ability.

[0117] RPIQ can provide more reliable performance evaluation when the data characteristics are not ideal by comparing the interquartile range of the true value of the data with the RMSEP of the model on the prediction set. The calculation formula is as follows:

[0118]

[0119] IQR usually refers to the interquartile range of the true value of the test set. The larger the RPIQ value, the better the model can effectively capture the regularity of the data, and the better the stability of the prediction results. Generally, when RPIQ>3, the model has high prediction accuracy; when RPIQ<1.5, the model prediction ability is relatively poor.

[0120] In this study, various classical pretreatment methods widely used in data analysis are systematically compared for possible problems in the original spectral data, including:

[0121] Wavelet denoising: using wavelet basis functions to perform multi-scale decomposition of spectral signals, effectively separating true signals and noise, preserving spectral details, and having excellent local feature capture ability.

[0122] Multiplicative scatter correction (MSC): This method is used to correct the multiplicative and additive effects of physical factors such as uneven sample particle size, different surface roughness, and changes in sample internal optical path, making different sample spectra more comparable.

[0123] Figure 3(a-e) respectively show the specific processing effect of the above five pretreatment methods on the original soil near-infrared spectroscopy data in this study.

[0124] To systematically evaluate the universality of T-SVAE feature extraction (i.e., the compatibility of features with multiple regression algorithms) and the robustness of the modeling process, this study designed a statistical performance analysis scheme based on repeated experiments. The specific process is as follows: 1) Use the trained T-SVAE to extract the latent mean vector from the preprocessed spectral data as a new feature set; 2) Combine this feature set with five regression models (RF, SVM, PLS, GBM, LR); 3) Use the same data set division scheme, and independently repeat the experiment 50 times for each combination; 4) Record the performance indicators of the test set for each experiment; 5) Calculate the average and standard deviation of the 50 experimental indicators, as shown in Table 1.

[0125] Table 1 Statistical performance analysis results of T-SVAE features combined with multiple regressors

[0126]

[0127] Table 1 summarizes the prediction results of T-SVAE features for classical linear (PLS, LR) and nonlinear (RF, SVM, GBM) regression models. From the experimental results, all five tested regression models show high prediction performance with the support of T-SVAE extracted latent features Among them, the nonlinear regression (0.982, 0.979, 0.987) effect is slightly better than the linear regression (0.966, 0.975) effect, but not significantly, indicating that the T-SVAE extracted features have strong adaptation ability and are less affected by linear or nonlinear regression mechanisms, and can well capture key encoding information from original, complex, and high-dimensional spectral data.

[0128] So far, the T-SVAE algorithm has shown excellent performance on the source data set used for its initial development and training, but whether it can be successfully applied to data sets of different sources and characteristics remains to be further tested, which is also the key to measuring its true portability and practical application value. For this purpose, this study selected three publicly available soil near-infrared spectroscopy data sets for testing.

[0129] To ensure the reproducibility of the proposed method and the fairness of the evaluation, a strict and uniform procedure was followed for the modeling of the external datasets. First, all the spectral data were pre-processed to eliminate instrumental and physical differences. Then, the pre-trained T-SVAE model was used to extract latent feature vectors from the pre-processed spectra. Finally, these high-quality features were input into the regression model for the final content prediction. The details of the pre-processing, regression modeling, and hyperparameter tuning for the three datasets are shown in Table 2. The performance evaluation results of the proposed T-SVAE model on these public datasets with different backgrounds are shown in Table 2.

[0130] Table 2. Evaluation of T-SVAE on soil properties from different regions

[0131]

[0132] The above results show that the T-SVAE model can still maintain high prediction accuracy and relatively stable performance when applied to these three external public datasets with different sources and characteristics. This strongly proves that the T-SVAE model does not simply learn some patterns or noise specific to its source dataset, but successfully captures more universal and fundamental spectral features and internal rules related to the potassium content in the soil. This robust performance in different soil types, different geographical regions, and potentially different experimental conditions fully demonstrates that the T-SVAE model has good cross-domain generalization ability.

[0133] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil, characterized in that, Includes the following steps: Step 1: Collect surface soil samples from different plots, obtain near-infrared spectral data of soil samples using a Fourier transform near-infrared spectrometer, and determine the actual content of available potassium in soil samples using the national standard method. Step 2: Preprocess the acquired near-infrared spectral data to remove impurity signals caused by instrument fluctuations, environmental interference, and differences in the physical morphology of the samples. Step 3: Construct a variational autoencoder model (T-SVAE) that integrates Transformer and supervised constraints. This model consists of an encoder, a latent space and sampling mechanism, a decoder, and a target regression output branch. Step 4: Set up a composite loss function that includes three types of loss terms, and perform joint optimization training on the T-SVAE model in an end-to-end manner; Step 5: Using the trained T-SVAE model encoder, perform forward propagation on the preprocessed near-infrared spectral data to extract the latent feature vector corresponding to each sample. Step 6: Input the extracted latent feature vector into the preset regression model to construct a soil available potassium prediction model, and output the soil available potassium content prediction result through the prediction model.

2. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 3, the encoder sequentially includes a batch normalization layer, a preliminary feature transformation layer, and multiple stacked TransformerBlocks. The batch normalization layer normalizes the input near-infrared spectral data to stabilize the network training process. The preliminary feature transformation layer uses a fully connected network combined with the LeakyReLU activation function to perform linear transformation and nonlinear mapping on the normalized spectral data. Each TransformerBlock internally sets up a multi-head self-attention mechanism and a feedforward fully connected network. The multi-head self-attention mechanism calculates the attention score between the query vector, key vector, and value vector to capture the global correlation between different dimensional features and wavelength points in the spectral sequence in parallel. The feedforward fully connected network performs nonlinear transformation on the output features of the attention mechanism, and the TransformerBlock internally promotes gradient flow through residual connections and layer normalization.

3. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 3, the latent space and sampling mechanism assume that the posterior probability distribution of the encoder output follows a Gaussian distribution, which is defined by the mean vector and log-variance vector of the encoder output. The latent variables are sampled from this Gaussian distribution through the reparameterization technique, so that the latent variables are differentiable with respect to the model parameters.

4. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 3, the decoder and encoder have a symmetrical design, which includes a preliminary feature transformation layer, multiple stacked TransformerBlocks and an output layer in sequence. The preliminary feature transformation layer uses a fully connected network combined with the LeakyReLU activation function to nonlinearly expand the latent variables.

5. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 4, the composite loss function includes reconstruction loss, KL divergence loss, and regression loss; reconstruction loss is used to measure the difference between the original near-infrared spectral data and the reconstructed spectral data.

6. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 2, the preprocessing method is selected from any one of wavelet transform, multivariate scattering correction, standard normal variable transformation, first derivative processing, and Savitzky-Golay smoothing. Wavelet transform separates the real signal from noise through multi-scale decomposition. Multivariate scattering correction and standard normal variable transformation eliminate the effects of optical path change and surface scattering. First derivative processing eliminates baseline drift and enhances the resolution of overlapping absorption peaks. Savitzky-Golay smoothing removes high-frequency random noise based on local polynomial least squares fitting.

7. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 4, the model training uses the Kennard-Stone algorithm to divide the training set and test set, with the training set accounting for 80% of the total samples and the test set accounting for 20%. The model hyperparameters are optimized using the Bayesian optimization method, including the number of fully connected network layers in the encoder and decoder, the number of neurons per layer, the number of attention heads in the Transformer module, the dimension of latent variables, the Dropout ratio, the learning rate, and the loss function hyperparameters. During training, an early stopping mechanism and a learning rate adjustment mechanism are enabled. The early stopping mechanism monitors the performance of the validation set, and terminates training if there is no improvement within a preset number of rounds. The learning rate adjustment mechanism reduces the learning rate by a preset factor when the performance of the validation set stagnates.

8. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, In step 6, the regression model is selected from any one of gradient boosting machine, support vector regression, random forest regression, partial least squares regression, and logistic regression. The regression model optimizes its hyperparameters through grid search and cross-validation, constructs the prediction relationship using the latent feature vectors extracted by the T-SVAE model, and outputs the prediction results of soil available potassium content. The prediction performance is evaluated by the coefficient of determination, root mean square error, mean absolute error, performance deviation ratio, and prediction interquartile range performance ratio.

9. The T-SVAE feature extraction strategy and its method for improving the accuracy of near-infrared spectroscopy measurement of available potassium in soil according to claim 1, characterized in that, After the trained T-SVAE model is combined with the regression model, it can be deployed on an edge computing device. The edge computing device must have at least 1GB of available storage, support the MicroPython environment and PyTorch, and be compatible with the ONNXRuntime and Triton inference framework.

Images