Causality method for judging hyperspectral band and substance content data

Abstract

The invention relates to a causality method for judging hyperspectral band and substance content data, which comprises the following steps: S1, acquiring a hyperspectral band X for a substance content Y, preprocessing the hyperspectral band X, and fixing the feature dimension of the preprocessed hyperspectral band to a first threshold number to obtain a preliminary hyperspectral data set Xprocess; s2, constructing a basic inversion model of the primary hyperspectral data set Xprocess and the substance content Y by using a catboostRegressor (catboostRegressor); s3, identifying an important related wave band range by using an SHAP model in combination with the basic inversion model and the initial hyperspectral data set Xprocess to obtain a hyperspectral data set XSHAP; s4, dividing the hyperspectral data set XSHAP into a training set and a test set according to a set proportion, estimating a conditional expectation function by using catboostRegressor, calculating a residual error by using the conditional expectation function, and calculating a causal effect value theta by using residual error regression; and S5, calculating an evaluation effect ATE based on the causal effect value theta. According to the method, the causal relationship between the hyperspectral wave band and the material content data is judged by fusing CatboostRegressor, SHAP and causal machine learning, and the causal relationship comprises correlation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and specifically to a method for determining the causality of hyperspectral band and material content data. Background Art

[0002] Hyperspectral imaging technology, capturing continuous nanoscale spectral signals (400–2500 nm) reflected from a material's surface, offers a revolutionary approach for nondestructive testing of material composition. Its core principle is that different materials (such as plant leaves, minerals, and soil) form unique reflectance spectral "fingerprints" in specific wavelengths due to properties such as molecular vibrations and electronic transitions. By mapping these spectra to material content (e.g., nitrogen, phosphorus, and potassium in soil, chlorophyll in plants) (i.e., "inversion"), it can replace traditional destructive chemical analysis and enable efficient, large-scale nondestructive testing in fields such as precision agriculture, environmental monitoring, and geological exploration.

[0003] Current hyperspectral inversion relies on a "data-driven" deep learning model, and its process is as follows: Data collection: The laboratory measures the actual content of the substance and simultaneously obtains the hyperspectral reflectance curve.

[0004] Model training: Use neural networks to learn the complex nonlinear relationship between spectrum (input) and content (output).

[0005] Interpretability supplement: Introduce interpretability tools such as SHAP to analyze band importance (correlation).

[0006] Fundamental flaws in existing approaches: Black-box dependency: The model only learns statistical correlations and lacks the embedding of physical mechanisms.

[0007] Weak interpretability: SHAP can identify “correlated bands” but cannot distinguish between correlation and causation. Summary of the Invention

[0008] To this end, an embodiment of the present invention provides a method for determining the causality of hyperspectral band and material content data, and provides a balanced, efficient, and accurate machine learning model, which cooperates with SHAP and causal machine learning (causalML) for algorithm fusion to ultimately express the causal coefficient between the band range and the material content.

[0009] A method for determining causality between hyperspectral band and material content data, comprising: S1. Collect hyperspectral band X for substance content Y, preprocess hyperspectral band X, fix the feature dimension of the preprocessed hyperspectral band to a first threshold, and obtain a preliminary hyperspectral dataset X_processed; S2. Use catboostRegressor to build a basic inversion model for the preliminary hyperspectral dataset X_processed and material content Y; S3. Use the SHAP model combined with the basic inversion model and the preliminary hyperspectral dataset X_processed to identify the important relevant band ranges and obtain the hyperspectral dataset X_SHAP; S4. Divide the hyperspectral dataset X_SHAP into training and test sets according to the set ratio, use catboostRegressor to estimate the conditional expectation function, use the conditional expectation function to calculate the residual, and use the residual regression to calculate the causal effect value θ; S5. Calculate the evaluation effect ATE based on the causal effect value θ.

[0010] Furthermore, the preprocessing includes standardizing or normalizing the hyperspectral band X, and then performing continuum removal and sliding window thinning.

[0011] Furthermore, the calculation formula of the SHAP value in S3 is as follows:

[0012] N is the set of all features (the first threshold band dimension); S is the feature subset, that is, each specific hyperspectral band; f is the basic inversion model, which is constructed by catboostRegressor in step S2.

[0013] Furthermore, in S4, the hyperspectral dataset X_SHAP is divided into X_train and X_test according to a set ratio, and the corresponding logistics content Y is divided into Y_train and Y_test, and T_train and T_test are provided for a certain characteristic spectral band; the use of catboostRegressor to estimate the conditional expectation function includes using catboostRegressor to learn and calculate g(X)=E[Y|X], inputting X_train and Y_train for learning, and obtaining the g(x) value by predicting X_test; and using catboostRegressor to learn and calculate m(X)=E[Y|X], inputting X_train and T_train for learning, and obtaining the m(x) value by predicting X_test.

[0014] Furthermore, the calculation formula for the causal effect value θ in S4 is as follows:

[0015] Where, Y is the outcome variable, i.e., the substance content value; T is the processing variable, a specific spectral band in the hyperspectral dataset X_SHAP, i.e., the target band; X is a high-dimensional covariate, the remaining bands in the hyperspectral dataset X_SHAP excluding the target band; g(X) and m(X) are the conditional expectation functions predicted by catboostRegressor.

[0016] Furthermore, the average treatment effect in S5 ; Among them, Y(1) and Y(0) represent the potential results of accepting and not accepting the treatment, respectively. Y(1) identifies the potential result when the treatment variable T=1, that is, the measured value of the substance content after turning on the specific spectral band; Y(0) identifies the potential result when the treatment variable T=0, that is, the measured value of the substance content after turning off the specific spectral band.

[0017] The embodiments of the present invention have the following advantages: This method combines SHAP correlation screening with DML causal validation, overcoming the limitations of traditional statistical methods. CatBoost provides predictive capabilities, while theta and ATE provide physical interpretation. By integrating CatboostRegressor, SHAP, and causal machine learning, the causal relationship between hyperspectral bands and material content data, including correlation, is determined. This technique further reduces the hyperspectral dimension (400nm-2500nm) to a combination of bands with actual causal relationships. Through double residual regression, it effectively alleviates the data shortage problem, significantly improving the efficiency of subsequent inversion models. Based on processing volume, it can achieve a 10-fold efficiency improvement without sacrificing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only exemplary. For those skilled in the art, other implementation drawings can be derived from the provided drawings without inventive effort. The structures, proportions, sizes, etc. illustrated in this specification are intended solely to facilitate understanding and reading by persons skilled in the art, and are not intended to limit the conditions under which the present invention may be implemented. Therefore, they have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, provided they do not affect the efficacy and objectives of the present invention, shall remain within the scope of the technical content disclosed in the present invention. Figure 1A flow chart of a method for determining causality between hyperspectral bands and material content data provided by an embodiment of the present invention; Figure 2 A flowchart of a method for using the SHAP model to identify important related band ranges in a method for determining the causality between hyperspectral bands and material content data provided by an embodiment of the present invention; Figure 3 A flow chart of a method for calculating ATE in a method for determining causality between hyperspectral bands and material content data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0020] like Figure 1 As shown, a method for determining the causality between hyperspectral band and material content data includes the following steps: S1. Collect hyperspectral band X for substance content Y, preprocess hyperspectral band X, and fix the feature dimension of the preprocessed hyperspectral band to a first threshold, such as 200, to obtain a preliminary hyperspectral dataset X_processed.

[0021] The pretreatment comprises: 1. Standardization / normalization: Perform $X_{\text{norm}} = \frac{X - \mu}{\sigma}$ or Min-Max normalization on each band.

[0022] 2. Continuum removal: Enhance absorption features $X_{\text{CR}} = \frac{X_{\text{norm}}}{\text{ConvexHull}(X_{\text{norm}})}$.

[0023] 3. Sliding window thinning.

[0024] S2. Use catboostRegressor to build a basic inversion model for the preliminary hyperspectral dataset X_processed and material content Y.

[0025] The regressor algorithm in the catboost series was used with parameters [“iterations”: 1000, “depth”: 7, “learning_rate”: 0.03, “l2_leaf_reg”: 7 ] to perform inversion training on the preliminary hyperspectral dataset X_processed and material content Y. The RMSE and R 2 To evaluate the model, the inversion training is injected into the entire dataset, that is, there is no division into training and test sets.

[0026] Basic inversion model content: Algorithm: CatBoostRegressor (Gradient Boosted Tree variant).

[0027] Parameters: {iterations:1000, depth:7, learning_rate:0.03, l2_leaf_reg:7}.

[0028] Training: Use the entire dataset (without splitting into training / test sets).

[0029] S3. Figure 2 As shown in the figure, the SHAP model is combined with the basic inversion model and the preliminary hyperspectral dataset X_processed to identify the important relevant band ranges and obtain the hyperspectral dataset X_SHAP.

[0030] The SHAP value in this technology is based on the Shapley value in game theory, and the formula is as follows:

[0031] N is the set of all features (the first threshold band dimension); S is the feature subset, that is, each specific hyperspectral band; f is the basic inversion model, which is constructed by catboostRegressor in step S2.

[0032] Filtering rules: 1 Select j > 0 (positive contribution); 2. Take the second threshold number of bands with the largest SHAP value, where the second threshold is less than the first threshold; 3 Output a new dataset: hyperspectral dataset X_SHAP.

[0033] Finally, the hyperspectral data with band dimensions whose ϕ is positive and ranks before the second threshold are screened out according to the SHAP algorithm to obtain the hyperspectral dataset X_SHAP.

[0034] S4. Use causalML combined with catboostRegressor method to verify the causal effect of important bands, divide the hyperspectral dataset X_SHAP into training set and test set according to the set ratio, use catboostRegressor to estimate the conditional expectation function, use the conditional expectation function to calculate the residual, and use the residual regression to calculate the causal effect value θ.

[0035] S41. Splitting the dataset The hyperspectral dataset X_SHAP is split into X_train and X_test in a 7:3 ratio. The corresponding logistics content Y is divided into Y_train and Y_test. For a specific spectral band, there are T_train and T_test.

[0036] S42. Estimating the conditional expectation function Use catboostRegressor to learn and calculate g(X)=E[Y|X], input X_train and Y_train for learning, and obtain the g(x) value by predicting X_test. The parameters are ["iterations": 1000, "depth": 5, "learning_rate": 0.05, "l2_leaf_reg": 5].

[0037] Use catboostRegressor to learn and calculate m(X)=E[Y|X], input X_train and T_train for learning, and obtain the value of m(x) by predicting X_test. The parameters are ["iterations": 1000, "depth": 5, "learning_rate": 0.05, "l2_leaf_reg": 5].

[0038] S43. Calculate residuals

[0039] That is: Y_tilde = Y_test - g(x)

[0040] That is: T_tilde = T_test - m(x) S44. Residual regression to calculate causal effect value θ Solve the causal effect θ based on the two equations. The larger it is, the greater the causality.

[0041]

[0042] Y is the outcome variable, i.e., the substance content value; T is the processing variable, a specific spectral band in the hyperspectral dataset X_SHAP, i.e., the target band; X is a high-dimensional covariate, the remaining bands in the hyperspectral dataset X_SHAP excluding the target band; g(X) and m(X) are the conditional expectation functions predicted by catboostRegressor; θ is the causal effect parameter; and η is the error term, satisfying E[ |X,T]=0 and E[η|X]=0.

[0043] The final causal effect value θ is as follows:

[0044] The meanings of the parameters in the formula are the same as above.

[0045] The causal effect value θ of each band is obtained. The band with a high causal effect value θ proves that it has both correlation and causality.

[0046] S5. Calculate the evaluation effect ATE based on the causal effect value θ like Figure 3 As shown, the average treatment effect (ATE) is: ; Y(1) and Y(0) represent the potential outcomes of accepting and not accepting the treatment, respectively.

[0047] Y(1) identifies the potential result when the treatment variable T=1, that is, the substance content measurement value after turning on a specific spectral band; Y(0) represents the potential result when the treatment variable T = 0, that is, the substance content measurement value after turning off the specific spectral band; The calculation and evaluation of ATE can express the accuracy of causal effect assessment and subsequent impact.

[0048] The estimation method of ATE of this technology evaluation effect is as follows:

[0049] Where ΔT is the amplitude of change in the band reflectivity (usually taken from the actual observation range).

[0050] The significance of the evaluation effect ATE of this technology is: the larger the ATE result, the stronger the causal effect, and the robust bands can be screened by combining the SHAP value correlation and causality.

[0051] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A method for determining the causality between hyperspectral bands and material content data, characterized in that: include: S1. Collect hyperspectral band X for substance content Y, preprocess hyperspectral band X, fix the feature dimension of the preprocessed hyperspectral band to a first threshold, and obtain a preliminary hyperspectral dataset X_processed; S2. Use catboostRegressor to build a basic inversion model for the preliminary hyperspectral dataset X_processed and material content Y; S3. Use the SHAP model combined with the basic inversion model and the preliminary hyperspectral dataset X_processed to identify the important relevant band ranges and obtain the hyperspectral dataset X_SHAP; S4. Divide the hyperspectral dataset X_SHAP into training and test sets according to the set ratio, use catboostRegressor to estimate the conditional expectation function, use the conditional expectation function to calculate the residual, and use the residual regression to calculate the causal effect value θ; S5. Calculate the evaluation effect ATE based on the causal effect value θ.

2. The method for determining causality between hyperspectral bands and material content data according to claim 1, characterized in that: The preprocessing includes standardizing or normalizing the hyperspectral band X, and then performing continuum removal and sliding window thinning.

3. The method for determining causality between hyperspectral bands and material content data according to claim 1, characterized in that: The calculation formula of the SHAP value in S3 is as follows: ； N is the set of all features (the first threshold band dimension); S is the feature subset, that is, each specific hyperspectral band; f is the basic inversion model, which is constructed by catboostRegressor in step S2.

4. The method for determining causality between hyperspectral bands and material content data according to claim 1, characterized in that: In the S4, the hyperspectral dataset X_SHAP is divided into X_train and X_test according to a set ratio, and the corresponding logistics content Y is divided into Y_train and Y_test. For a certain characteristic spectral band, there are T_train and T_test; The method of using catboostRegressor to estimate the conditional expectation function includes using catboostRegressor to learn and calculate g(X)=E[Y|X], inputting X_train and Y_train for learning, and obtaining the value of g(x) by predicting X_test; And use catboostRegressor to learn and calculate m(X)=E[Y|X], input X_train and T_train for learning, and obtain the m(x) value by predicting X_test.

5. The method for determining causality between hyperspectral bands and material content data according to claim 1, characterized in that: The calculation formula of the causal effect value θ in S5 is as follows: ； Where, Y is the outcome variable, i.e., the substance content value; T is the processing variable, a specific spectral band in the hyperspectral dataset X_SHAP, i.e., the target band; X is a high-dimensional covariate, the remaining bands in the hyperspectral dataset X_SHAP excluding the target band; g(X) and m(X) are the conditional expectation functions predicted by catboostRegressor.

6. The method for determining causality between hyperspectral bands and material content data according to claim 1, characterized in that: The average treatment effect ATE=E[Y(1)−Y(0)] in S5; Among them, Y(1) and Y(0) represent the potential results of accepting and not accepting the treatment, respectively. Y(1) identifies the potential result when the treatment variable T=1, that is, the substance content measurement value after turning on the specific spectral band; Y(0) identifies the potential result when the treatment variable T=0, that is, the substance content measurement value after turning off the specific spectral band.

Images