Rare earth element content quantitative estimation method, device and equipment and storage medium

Abstract

The invention provides a rare earth element content quantitative estimation method and device, equipment and a storage medium. The method relates to the technical field of hyperspectrum, machine learning and quantitative estimation. The method comprises the following steps: acquiring hyperspectral data of each sample, eliminating gross error points, performing data dimension reduction and K-means clustering analysis to obtain effective spectral data of each sample, performing denoising processing, averaging to obtain an average spectral curve, and extracting spectral characteristics; analyzing the correlation between the characteristic data of each wave band and the element content by utilizing Pearson correlation, selecting a high-correlation wave band range, carrying out importance test on the high-correlation wave band range by utilizing a random forest, and screening out a characteristic wave band corresponding to each element; and constructing a data set by using the characteristic wave bands corresponding to the elements and the chemical test contents of the elements, and training and evaluating the machine learning model based on the data set to obtain a quantitative estimation model. The rare earth element content can be quickly scanned and evaluated in time.

Description

Technical Field

[0001] The present application relates to the fields of hyperspectral, machine learning and quantitative estimation technology, and in particular to a method, device, equipment and storage medium for quantitative estimation of rare earth element content. Background Art

[0002] Currently, deep-sea rare earth exploration primarily relies on a combination of geophysical methods, such as sediment core sampling and shallow stratigraphic profiling. Rare earth content is determined through core sample testing, combined with shallow stratigraphic data for resource assessment. With the introduction of deep-sea towed bodies, ROVs, AUVs, and HOVs, research is also underway to obtain in situ sediment parameters using in situ testing. Laboratory elemental content testing using samples and in situ testing using a transport platform share the commonality of stationary testing, characterized by relatively accurate positioning and high-precision laboratory measurement data. However, stationary sampling and testing, performed at point locations, results in low precision in the lateral spatial distribution of data, long average processing time, and high economic costs. This facilitates research and verification determinations but hinders surface detection and rapid assessment.

[0003] Hyperspectral detection technology, with its large information capacity, high resolution, and rapid scanning speed, has been widely used in fields such as agriculture, environmental monitoring, and mineral exploration. Its integration with drone technology has further expanded its application. In agriculture, it can assess the effective substance content of crops, chlorophyll content, and nutrients needed for crop growth. In the environmental field, it is often used for hyperspectral inversion of heavy metal ions in soil and salinized land. In mineral exploration, hyperspectral analysis has demonstrated significant application value, enabling the estimation of rare earth element content and resource assessment, providing a fast, economical, and efficient means for obtaining and evaluating rare earth content.

[0004] There are various methods for estimating elemental content using hyperspectral data, such as partial least squares regression (PLSR), support vector machine regression (SVR), random forest (RF), and extreme learning machine (ELM). These methods have been successfully applied in fields such as terrestrial mineral resource assessment and ecological environment research. Machine learning methods can effectively invert the geochemical composition of rocks. Trace element content can also be indirectly estimated by leveraging inter-element relationships. In recent years, new modeling techniques have been applied to elemental content estimation, such as self-attention neural network (SCSANet), deep learning, and geographically weighted gradient boosting (GW-XGBoost), effectively enabling the inversion of multiple metal element contents. Feature extraction and characteristic band selection are particularly important in constructing models for estimating target component content. Hyperspectral feature extraction methods include first-order derivative (FD), second-order derivative (SD), multivariate scatter correction (MSC), and continuum removal (CR). Different feature extraction methods (such as correlation analysis and absorption peak depth) vary in the wavelength range and number of characteristic bands selected, resulting in different estimation results and effectiveness. Some studies suggest that using derivatives for feature extraction can better reveal hidden information in spectral curves and achieve better estimation results. Currently, ocean hyperspectral technology is primarily used to estimate foreign matter such as oil spills on the sea surface, chlorophyll parameters, and phytoplankton. Underwater hyperspectral imaging, which offers rich spectral information and high resolution, has gradually gained momentum in recent years. Underwater robotic technologies, such as tethered underwater vehicles (ROVs) and autonomous underwater vehicles (AUVs), provide the foundation for rapid, high-precision near-seafloor exploration. These technologies can be equipped with hyperspectral equipment for high-precision exploration near the seafloor, providing a technical solution for the optical detection of seafloor polymetallic nodules, crusts, rare earth elements, sulfides, and other mineral resources.

[0005] Currently, there is no reported method for estimating the rare earth element content in marine sediments using hyperspectral technology. Therefore, how to estimate the rare earth element content in marine sediments using hyperspectral technology with high accuracy, large information carrying capacity and fast acquisition speed is the technical problem to be solved in this application. Summary of the Invention

[0006] This application provides a method, apparatus, device, and storage medium for quantitatively estimating rare earth element content in marine sediments. This method addresses the feasibility of estimating rare earth element content in marine sediments using hyperspectral technology, which offers high precision, large information capacity, and rapid acquisition speed. Using marine sediments as the test and research object, elemental chemical data of the samples is acquired using inductively coupled plasma mass spectrometry (ICP-MS). Hyperspectral data of the samples under the same ICP-MS conditions is obtained using an imaging spectrometer with a wavelength range of 380-1000 nm and a spectral resolution of 2.5 nm. Based on this, the spectral data is preprocessed and analyzed to investigate the impact of varying surface flatness, acquisition angles, degrees of looseness, and local shadow conditions on the spectral information of the same sample. Spectral feature extraction, elemental characteristic spectral band screening, and quantitative estimation models are developed. The effectiveness of the methods is compared during the analysis, processing, and estimation process, and the prediction accuracy and effectiveness of the models are evaluated. Experimental results and conclusions are presented to provide a reference for the application of hyperspectral technology in estimating rare earth element content in sediments and other scenarios.

[0007] In a first aspect, the present application provides a method for quantitatively estimating rare earth element content, comprising:

[0008] Acquire hyperspectral data of each sample; wherein the hyperspectral data includes a plurality of data points, and each data point includes a plurality of spectral band data;

[0009] Based on the hyperspectral data of each sample, large error points are eliminated by quartiles to perform data dimensionality reduction, and the data features after dimensionality reduction are subjected to K-means cluster analysis to obtain the effective spectral data of each sample, and the effective spectral data of each sample are denoised and averaged to obtain an average spectral curve;

[0010] Extracting spectral features based on the average spectral curve; wherein the spectral features include characteristic data of each band;

[0011] The correlation between the characteristic data of each band and the element content was analyzed using Pearson correlation, and the highly correlated band range was selected. The importance of the highly correlated band range was tested using random forest to obtain important characteristic bands and screen out the characteristic bands corresponding to each element.

[0012] A data set is constructed by combining the characteristic bands corresponding to each element and the element's chemical test content, and the data set is divided into a training set and a prediction set. The training set is used to train multiple machine learning models; wherein the multiple machine learning models include a partial least squares regression model, an extreme gradient boosting model, a multilayer perceptron, a support vector machine regression model, and a random forest model;

[0013] The prediction set was predicted using multiple trained machine learning models, and compared with the actual measurement values. The determination coefficients, relative performance differences, root mean square errors, and mean absolute percentage errors of the trained machine learning models were calculated to evaluate the model accuracy. The trained machine learning model with the highest model accuracy was selected as the quantitative estimation model, which was used to achieve hyperspectral quantification of rare earth element content in marine sediments.

[0014] In a possible design, based on the hyperspectral data of each sample, large error points are eliminated by quartiles, and the data dimension reduction is performed in the following manner:

[0015] The acquired hyperspectral data are synthesized into a matrix and standardized using the following formula:

[0016]

[0017] Where, σ i is the standard deviation of the i-th sample, X i is the random variable of the i-th sample, is the sample mean of the i-th sample, x i is the feature of the i-th sample in the standardized sample matrix;

[0018] The covariance matrix R of the standardized sample matrix is ​​calculated as follows:

[0019]

[0020] Where, X ′ is the standardized data matrix, T is the matrix transpose, and N is the number of samples;

[0021] Calculate the eigenvalues ​​and eigenvectors of the covariance matrix R, where the eigenvectors represent the directions of the principal components and the magnitude of the eigenvalues ​​represents the importance of each principal component. Select the first m eigenvectors with the largest eigenvalues ​​as the principal components to obtain the data features after dimensionality reduction.

[0022] In one possible design, the data features after dimensionality reduction are subjected to K-means cluster analysis to obtain valid spectral data of each sample in the following ways:

[0023] Randomly select K data points as initial cluster centers;

[0024] Assignment: Calculate the distance between each data point and each cluster center, and assign the data point to the cluster center closest to it;

[0025] Update: For each cluster center, after adding a new data point, the mean of all data points in the cluster is used as the new cluster center;

[0026] The allocation and updating are repeated until a set condition is met, wherein the set condition includes reaching a set number of iterations.

[0027] In a possible design, based on the average spectral curve, ways of extracting spectral features include normalization, first-order derivative, multivariate scatter correction, continuum removal, logarithmic first-order derivative, and reciprocal logarithmic first-order derivative.

[0028] In one possible design, the correlation between the characteristic data of each band and the element content is analyzed using Pearson correlation. The methods for selecting the high-correlation band range include:

[0029] The correlation coefficient is calculated using the following formula:

[0030]

[0031] Where r j is the correlation coefficient of the jth band, x i Indicates the content of an element, y i represents the spectral characteristic value under the j-th band, and Indicates the sample mean of the two sets of data analyzed in this band;

[0032] Setting a correlation coefficient threshold, and determining a band with a correlation coefficient greater than the correlation coefficient threshold as a high-correlation band;

[0033] The maximum value of all high-correlation bands is increased by 10 nm as the rightmost band range, and the minimum value is reduced by 10 nm as the leftmost band range to obtain the high-correlation band range.

[0034] In a possible design, random forest is used to perform importance testing on the highly correlated band range to obtain important characteristic bands. Methods for screening characteristic bands corresponding to each element include:

[0035] Inputting the highly correlated band range into a random forest regression model, wherein the random forest regression model takes element content as a dependent variable and band spectrum value as a feature;

[0036] The importance of each band is calculated using the following formula:

[0037]

[0038] Where, ΔImpurity n1 (f) is the purity reduction value of the feature band f at node n1, T1 is the total number of trees, I(f) represents the importance of the band, which is the average impurity reduction of the feature band f in all trees, t is the ordinal number of the tree, and n1 is the node of the tree;

[0039] According to the importance of each band, the characteristic bands corresponding to each element are screened out.

[0040] In one possible design, the characteristic bands corresponding to each element are screened out based on the importance of each band, including:

[0041] Test the first m most important bands; if all pass the significance test, expand the range to the first m + m1 bands, and so on, until an insignificant band appears, and obtain the characteristic band corresponding to each element; where m and m1 are set constants.

[0042] In a second aspect, the present application provides a device for quantitatively estimating rare earth element content, the device comprising:

[0043] A data acquisition module is configured to acquire hyperspectral data of each sample; wherein the hyperspectral data includes a plurality of data points, and each data point includes a plurality of spectral band data;

[0044] The data preprocessing module is configured to perform data dimension reduction based on the hyperspectral data of each sample by eliminating large error points through quartiles, perform K-means cluster analysis on the data features after dimension reduction to obtain effective spectral data of each sample, and perform denoising on the effective spectral data of each sample and average the data to obtain an average spectral curve;

[0045] A spectral feature extraction module is configured to extract spectral features based on the average spectral curve; wherein the spectral features include characteristic data of each band;

[0046] The characteristic band screening module is configured to use Pearson correlation to analyze the correlation between the characteristic data of each band and the element content, select a highly correlated band range, and use random forest to perform an importance test on the highly correlated band range to obtain important characteristic bands, and screen out the characteristic bands corresponding to each element;

[0047] A model training module is configured to construct a data set based on the characteristic bands corresponding to each element and the element's chemical test content, divide the data set into a training set and a prediction set, and use the training set to train multiple machine learning models; wherein the multiple machine learning models include a partial least squares regression model, an extreme gradient boosting machine model, a multilayer perceptron, a support vector machine regression model, and a random forest model;

[0048] The model evaluation module is configured to use multiple trained machine learning models to predict the prediction set, compare the actual measurement values, calculate the determination coefficient, relative performance difference, root mean square error and mean absolute percentage error of the multiple trained machine learning models, evaluate the model accuracy, and select the trained machine learning model with the highest model accuracy as the quantitative estimation model, which is used to achieve hyperspectral quantification of rare earth element content in marine sediments.

[0049] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for quantitative estimation of rare earth element content as described in the first aspect and various possible designs of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for quantitatively estimating the rare earth element content as described in the first aspect and various possible designs of the first aspect is implemented.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for quantitatively estimating the rare earth element content as described in the first aspect and various possible designs of the first aspect.

[0052] The method, device, equipment and storage medium for quantitative estimation of rare earth element content provided in this application have at least the following features:

[0053] Beneficial effects:

[0054] Although factors such as the surface unevenness and different looseness of the sediment will affect the quality fluctuation of a small part of the data, this application can eliminate these factors by removing large error points through quartiles, performing data dimensionality reduction, and performing K-means cluster analysis and noise reduction averaging on the data features after dimensionality reduction, without affecting the application of the data.

[0055] This application uses Pearson correlation to analyze the correlation between the characteristic data of each band and the element content, selects the highly correlated band range, and uses random forest to perform an importance test on the highly correlated band range to obtain important characteristic bands and screen out the characteristic bands corresponding to each element. It can accurately lock the characteristic bands, reduce the impact of redundant information, and improve calculation efficiency and estimation accuracy.

[0056] This application uses the reciprocal logarithm first-order reciprocal feature extraction method and the multi-layer perceptron model to effectively extract most of the rare earth element spectral features and obtain a good estimation result of the rare earth element content, especially the light rare earth element content.

[0057] The results of quantitative estimation of rare earth element content show that the relative performance difference of light rare earth element estimation results using hyperspectral technology is greater than 2.00, and the number of characteristic bands is between 7 and 17. Experiments have shown that hyperspectral technology can be used to carry out quantitative estimation of rare earth element content in marine sediments, providing reference and technical ideas for the detection and estimation of marine rare earth and various metal minerals. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 A flowchart of a method for quantitatively estimating rare earth element content provided in an embodiment of the present application;

[0060] Figure 2 Schematic diagram of the experimental environment setup and sample preparation provided in the examples of the present application; (a) ICP-MS chemical testing of sediment samples; (b) sample preparation before hyperspectral testing (samples are spread flat in experimental containers, and all experimental containers containing samples are placed on a D-shaped tube); (c) hyperspectral data acquisition under natural light (sediment is placed on a black background cloth, and a line scan hyperspectral camera is connected to a slide rail, and hyperspectral data of the sediment sample is acquired through the uniform motion of the slide rail);

[0061] Figure 3 This is a sample spectral point clustering result diagram provided in the embodiment of the present application;

[0062] Figure 4 This is a stability analysis diagram of different types of spectral data points after clustering provided in an embodiment of the present application;

[0063] Figure 5 The average spectral curve of each sample after the selected points provided in the embodiment of the present application;

[0064] Figure 6 This is a diagram of the MLP regression model structure provided in the embodiment of the present application;

[0065] Figure 7Correlation results between the preprocessing results of different types of points and element content provided in the embodiment of the present application (taking La as an example); (a) normalization (NOR); (b) first-order derivative (FD); (c) multivariate scatter correction (MSC); (d) continuum removal (CR); (e) logarithmic first-order derivative (LOG-FD); (f) reciprocal logarithmic first-order derivative (TLOG-FD);

[0066] Figure 8 Results of different feature extraction methods provided in the embodiments of the present application (taking La as an example); among them, (a) normalization (NOR); (b) first-order derivative (FD); (c) multivariate scatter correction (MSC); (d) continuum removal (CR); (e) logarithmic first-order derivative (LOG-FD); (f) reciprocal logarithmic first-order derivative (TLOG-FD);

[0067] Figure 9 This is a graph showing the results of characteristic band screening and analysis of elements based on different feature extraction methods provided in the embodiments of the present application (taking La as an example);

[0068] Figure 10 Taylor diagram comparing different feature extraction methods and prediction results of different models provided in the embodiments of the present application;

[0069] Figure 11 This is a structural diagram of the device for quantitatively estimating rare earth element content provided in an embodiment of the present application.

[0070] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0071] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0072] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0073] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0074] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0075] Marine rare earth-rich sediments contain significant quantities of rare earth elements, with medium and heavy rare earth elements presenting far higher concentrations than on land, demonstrating significant resource potential. Current exploration and assessment methods for deep-sea rare earths rely primarily on geological column sampling and laboratory testing. However, rapid and comprehensive assessment of rare earth element content is difficult. Hyperspectral technology, with its ability to rapidly and comprehensively measure elemental content with high precision, holds great potential for resource assessment. Based on this, an embodiment of the present application provides a method for quantitative estimation of rare earth element content. This embodiment takes marine sediments as the research object, obtains chemical data of 53 uniformly ground sediment samples and their 380-1000nm hyperspectral data under natural light conditions, analyzes the influence of sample surface flatness and shadow occlusion factors on the characteristics of the acquired spectral data, and performs data optimization and different spectral feature extraction methods (normalization, first-order derivative, multivariate scattering correction, continuum removal, logarithmic first-order derivative, inverse logarithmic first-order derivative) on the hyperspectral data. The feature importance test method within the high correlation range band is used to determine the characteristic bands of each rare earth element, and different estimation models are trained to estimate the rare earth element content in marine sediments.

[0076] like Figure 1 , which is a flow chart of a method for quantitatively estimating rare earth element content provided in an embodiment of the present application, the method for quantitatively estimating rare earth element content includes the following steps S100 - S600 .

[0077] S100: Acquire hyperspectral data of each sample; wherein the hyperspectral data includes a plurality of data points, and each data point includes a plurality of spectral band data.

[0078] In this example, 53 marine surface sediment samples were selected as test objects. Their distribution mainly included surface sediment samples from the waters near Antarctica (70°-78°E, 64°-69°S), and surface sediment samples from the Bering Sea and the Chukchi Sea (167°-180°W, 56°-72°N). The water depth was approximately 30-3500 m, and the water depth conditions, sediment sources, and geological environment backgrounds were all different.

[0079] Rare earth element content was determined using inductively coupled plasma-mass spectrometry (ICP-MS). ICP-MS combines the generation capabilities of inductively coupled plasma (ICP) with the separation capabilities of mass spectrometry (MS). ICP converts samples into ions, which are then measured using a mass spectrometer to determine elemental content. The statistical results for the rare earth elements in the samples are shown in Table 1.

[0080] The data showed high concentrations of light rare earth elements (LEEs), such as La, Ce, Pr, and Nd, with Ce averaging 58.37 mg / kg. The concentrations of medium and heavy REEs were relatively low, while the concentrations of heavy REEs Sc and Y were relatively high. The total REE content averaged 160.67 mg / kg. The coefficient of variation (CV, standard deviation / mean) for each REE was less than 1, indicating that the spatial distribution of REEs is random and uniform, and is minimally affected by human activities.

[0081] Table 1. Statistical analysis of rare earth element content in marine surface sediment samples

[0082]

[0083] (Note: * stands for ppm)

[0084] The first step in collecting sample hyperspectral data is to prepare the hyperspectral experimental sample. Fix the experimental container with a D-type tube, keep the surface height consistent, and mark the sample numbers in sequence. Take a portion of each sample that has been tested for chemical element content, place the corresponding sample number in it, and lay it out naturally. Figure 2 As shown in (b).

[0085] In order to study the ability and effectiveness of hyperspectral technology in identifying and estimating sediment elements in sediment samples close to natural state and environment, the experimental environment was set up and the sample preparation (such as Figure 2 (as shown in (b)), the test process maintains a natural test environment and a quasi-natural test target. Environmental conditions include: ambient noise, shadows (between containers or samples), complex test environments (affected by ambient light), instrument carrier slider jitter, and changes in acquisition distance. Test target conditions include: varying surface looseness and aggregation between each sediment sample, varying surface flatness, slightly varying sample dryness and humidity, and a non-single target (the background material in the D-tube consists of sediment of varying particle sizes, mixed with micronodules, sand, and calcareous fossils).

[0086] Hyperspectral data were collected using a line-scan hyperspectral camera with a measurement band range of 380-1000 nm, including spectral data of 480 bands, a spectral resolution of 2.5 nm, a camera field of view of 70°, and 1200 pixels in the vertical direction. Before the experiment, the sample prepared in 2.1.3 was placed under a dark background under natural light conditions, such as Figure 2 As shown in (c) in the figure. During the test, the slide rail for vertically fixing the hyperspectral camera is placed above the sample, as shown in Figure 2 As shown in (c), the slide speed is set to 20 mm / s, the camera frame rate is set to 28 fps, and the camera exposure time is 5110, so that the maximum radiance obtained does not exceed the upper limit of the light intensity radiance that the instrument can obtain. The distance between the hyperspectral camera lens and the sample is set at different distances: 0.15 m, 0.68 m, and 1.00 m, respectively. The data accuracy obtained is approximately 69, 15, and 10 pixels per centimeter. Too high a spectral pixel count will result in a large amount of data, while too low a spectral pixel count will result in serious edge mixing of the spectrum. Therefore, this embodiment comprehensively considers the use of hyperspectral data obtained at 0.68 m as the original spectral data. The data point density obtained for each sample in the experiment is greater than 20×20 (the sample container inner diameter is 25 mm), and a total of 480 spectral band data are obtained for each data point. The highest radiance value is approximately between 1000 and 3000, and the lowest is less than 20. The low values ​​are mainly distributed at the front and end of the spectral data.

[0087] S200: Based on the hyperspectral data of each sample, large error points are eliminated through quartiles to perform data dimensionality reduction, and the data features after dimensionality reduction are subjected to K-means cluster analysis to obtain the effective spectral data of each sample, and the effective spectral data of each sample are denoised and averaged to obtain an average spectral curve.

[0088] When collecting hyperspectral data in a natural environment, it is generally affected by the influence of ambient scattered light, instrument stability, and shadows caused by the angle of the natural light source. In addition, the natural surface state retained during the sample preparation process (different flatness, different polymerization degrees of powder samples) will also cause spectral reflection and scattering effects. This kind of environmental scattering, instrument stability, light source angle, shadow, target surface flatness and looseness, etc., are close to the natural collection state of dynamic data acquisition carried out by hyperspectral detection equipment. Therefore, it is necessary to first eliminate highly abnormal data affected by environmental scattering, instrument stability and other data collection aspects. Then, the validity of the hyperspectral data obtained under the influence of the carrying conditions is tested.

[0089] In this embodiment, the quartile method is first used to eliminate the obviously abnormal spectral point data of the surface spectral points of each sample, and then PCA dimensionality reduction and K-means cluster analysis are performed to test and screen the effective spectral data of each sample.

[0090] In some embodiments, data dimensionality reduction is performed using PCA. The principal components analysis (PCA) method calculates variance and uses the direction with the largest data variance as the basis (or principal component) of the new coordinate system. This reduces the data dimension, eliminates redundancy, simplifies the data set, and retains important data information as much as possible. The steps are as follows:

[0091] S201, synthesize the acquired hyperspectral data into a matrix where N is the number of samples (in this embodiment, N=53), p is the number of feature dimensions, and the random variables X1, X2, ..., X P , the sample mean is The sample standard deviation is denoted as σ1, σ2,…, σ P , perform standardization transformation,

[0092]

[0093] Where, σ i is the standard deviation of the i-th sample, X i is the random variable of the i-th sample, is the sample mean of the i-th sample, x i is the feature of the i-th sample in the standardized sample matrix.

[0094] S202, calculate the covariance matrix of the standardized sample matrix, X ′ is the standardized data matrix,

[0095]

[0096] Where, X ′ is the standardized data matrix, T is the matrix transpose, and N is the number of samples.

[0097] S203, solving eigenvalues ​​and eigenvectors: Calculate the eigenvalues ​​and eigenvectors of R. The eigenvectors represent the directions of the principal components, and the eigenvalues ​​represent the importance of each principal component. Select the first m eigenvectors with the largest eigenvalues ​​as the principal components to complete data dimensionality reduction.

[0098] In some embodiments, the specific steps of performing K-means cluster analysis on the data features after dimensionality reduction to obtain the effective spectral data of each sample are as follows:

[0099] S210, randomly select K data points as initial cluster centers.

[0100] S220, for each data point in the data set, calculate its distance from each cluster center and assign it to the cluster center closest to it, usually using Euclidean distance as the distance metric.

[0101] S230: For each cluster, recalculate its cluster center. The new cluster center is the mean of all data points in the cluster.

[0102] S240, Iteration: Repeat the allocation and update steps until the termination condition is met.

[0103] This example analyzes hyperspectral data and processes it using the aforementioned method. The 53 samples collected from the hyperspectral data were all uniform powders. Using the aforementioned acquisition method, approximately 400 points of data were obtained for each sample. Each point contained 480 radiance data points, meaning that one chemical data point corresponds to approximately 192,000 (400 × 480) spectral data points.

[0104] First, the error of the hyperspectral data of each sample was evaluated, and the quartile method was used to eliminate outliers with greater influence to ensure the repeatability and credibility of the results; then, the PCA method was used to extract important spectral features, and the first three dimensions after dimensionality reduction (data features >90%) were selected to ensure that data features were not lost, while greatly reducing the amount of data and improving computational efficiency; finally, K-means clustering was used to perform unsupervised cluster analysis based on the spectral data of each sample after dimensionality reduction. Experiments showed that when the cluster center was set to 4, shadow features and the loose state of sediments could be effectively distinguished.

[0105] After repeated experiments, it was shown that when the cluster center is set to 4, the shadow features and the loose state of the sediment can be effectively distinguished. The classification results are as follows: Figure 3 As shown. KS4 (Class 4) has the highest and most stable spectrum intensity, corresponding to Figure 2 In (b), the surface of the sample is smooth and compact, and the spectral intensity of ks3 (type 3) is second, corresponding to Figure 2 In the loose part of the sample (b), the ks2 (type 2) spectrum intensity is more discrete, corresponding to Figure 2 Middle (b) shows the relatively loosely packed portion of the sample. The spectral intensity dispersion of KS1 (category 1) is the greatest and most unstable, often due to significant shadowing or mixed spectra with the container. Data stability decreases from category 4 to category 1.

[0106] After polynomial fitting of the standard deviation and coefficient of variation results of each band, the standard deviation and coefficient of variation of each band also showed corresponding changes.

[0107] like Figure 4As shown, the standard deviation of the KS1 points is approximately 0-5 points higher than that of the other three categories. The differences in coefficient of variation are small, with the KS1 point coefficient slightly higher than that of the other three categories. This instability of KS1 is primarily due to the influence of shadow features and edge-mixed spectra, which significantly impacts the estimation results. To maximize the effectiveness of spectral points under different conditions, we selected categories 1, 2, 3, and 4, 2, 3, and 4, and 3, 4, and 4 as the final sample points based on data stability. We obtained the average spectral curve corresponding to each sample, performed feature extraction, and conducted correlation analysis with the corresponding sample element content. The results showed that all data types were representative and usable.

[0108] Figure 4 The results also show that the coefficient of variation of the bands at the front and back ends of the data collection is high, and has a greater randomness. The coefficient of variation of the bands before 410nm and after 900nm is greater than 30%, which has a strong randomness. Therefore, the bands before 410nm are eliminated. However, considering that there are many bands after 900nm and there may be characteristic bands of related elements, the bands after 945nm with a coefficient of variation greater than 50% (with extremely strong randomness) are eliminated.

[0109] In order to ensure and highlight the experimental effect of the subsequent estimation method, the ks1 data affected by shadows and mixed spectra were eliminated, and the more stable ks2, ks3, and ks4 spectral data were finally selected as input data. The average spectral curve of each sample is shown in Figure 2. Figure 5 shown.

[0110] S300: Extracting spectral features based on the average spectral curve; wherein the spectral features include characteristic data of each band.

[0111] After obtaining the sample's average spectral curve, dark current and high-frequency noise still exist. Therefore, this example further performs dark current correction on the data and removes noise using a Savitzky-Golay smoothing filter. SG smoothing effectively eliminates high-frequency components and noise while maintaining the shape and width of the spectral signal. In this example, the spectral signals acquired by the front-end and back-end bands of the hyperspectral camera are weak and have a low signal-to-noise ratio. Based on the above analysis results, the bands before 410nm and after 945nm are removed and not analyzed in subsequent processing.

[0112] In some embodiments, considering that different minerals have different spectral characteristics and require different feature extraction methods, a suitable feature extraction method can more accurately extract spectral features and reduce redundant bands, which is key to the subsequent establishment of an estimation model. Therefore, this embodiment uses six typical band feature extraction methods: normalization (NOR), first-order derivative (FD), multivariate scatter correction (MSC), continuum removal (CR), logarithmic first-order derivative (LOG-FD), and reciprocal logarithmic first-order derivative (TLOG-FD) to extract spectral features. The NOR method normalizes spectral radiance data to the range of 0 to 1, allowing data to be compared at the same data level. The MSC method easily eliminates baseline drift and amplitude changes caused by differences in physical properties by calculating the average of all measured spectra as the baseline spectrum and performing linear regression correction on each measured spectrum, thereby eliminating irrelevant changes. The CR method can enhance spectral absorption characteristics. After removing the continuum, the radiance data is normalized to the range of 0 to 1, which is beneficial for comparison with other spectral curves, extracting characteristic bands, and better performing element estimation. The FD method can enhance weak spectral information, help to distinguish spectral differences between different substances, and facilitate the extraction of absorption peak information in the spectrum. The LOG-FD and TLOG-FD methods first perform logarithmic transformation to reduce heteroscedasticity, compress the data size, and make the data more stable, and then perform first-order derivative feature extraction to highlight weak information.

[0113] In this embodiment, the pre-processed spectral data is processed by the above method, and the results and corresponding estimation effects are analyzed. Adjustments and experiments are repeated to finally select a spectral feature extraction method suitable for rare earth elements in sediments.

[0114] S400: Analyze the correlation between the characteristic data of each band and the element content using Pearson correlation, select the highly correlated band range, and perform an importance test on the highly correlated band range using random forest to obtain important characteristic bands, and screen out the characteristic bands corresponding to each element (PCCR-FI).

[0115] The selection of characteristic bands is crucial to the scientific and rationality of sediment REE estimation results. High correlations between characteristic bands (or specific bands) and sediment REE content in the sample's hyperspectral data after spectral feature extraction indicate the potential for further extraction of characteristic bands and the construction of corresponding estimation models for elemental content assessment. However, since similar minerals or elements of different valence states coexist in nature, resulting in similar spectral signatures, directly using highly correlated bands with chemical data to build models can result in lower accuracy.

[0116] To address this issue, this embodiment, based on step S400, utilizes correlation analysis to screen effective wavelength ranges, combined with significance testing to screen for characteristic wavelengths of rare earth elements in sediments. This method first selects highly correlated wavelength ranges and ranks them for significance. This method then obtains a set of wavelengths within these highly correlated wavelength ranges that exhibit absorption peaks with high correlations and FI indices. This set of wavelengths is then tested for significance to eliminate any chance events due to small sample sizes. In this embodiment, wavelengths with a significance test result of p < 0.01 are selected as characteristic wavelengths for the element.

[0117] The correlation analysis formula is as follows:

[0118]

[0119] Among them, r j is the correlation coefficient of the jth band, x i Indicates the content of an element, y i represents the spectral characteristic value under the j-th band, and In this embodiment, i=1, 2, .... 480; n=53.

[0120] After performing correlation analysis and significance testing, this embodiment uses random forest analysis to analyze the band importance of highly correlated bands to determine the characteristic bands. The random forest importance test measures the importance of each characteristic band by training a random forest model containing multiple decision trees and calculating the average node impurity reduction of the feature in the node splits of all trees. The importance assessment is the mean decrease in impurity (MDI), which is calculated as follows:

[0121]

[0122] Where, ΔImpurity n1 (f) is the purity reduction value of the feature band f at node n1, T1 is the total number of trees, I(f) represents the importance of the band, which is the average impurity reduction of the feature band f in all trees, t is the ordinal number of the tree, and n1 is the node of the tree.

[0123] This embodiment is a regression problem analysis, and its impurity index is mainly represented by mean square error. The reduced impurity of the feature is calculated each time a node is split. These reductions are averaged across all trees to obtain the importance of the feature band.

[0124] This embodiment uses the PCCR-FI method to screen characteristic bands. During implementation, the high correlation band range is first determined based on the correlation and significance test results. In order to fully cover the extremely significantly correlated bands and prevent the characteristic bands from being missed, the adjacent band range is set according to the absorption peak of the screened high correlation band. The adjacent band range of this embodiment is determined by increasing the maximum value of the band with the extremely significantly correlated band and the absolute value of the correlation greater than 0.1 by 10nm as the rightmost band range, and reducing the minimum value of the band by 10nm as the leftmost band range. All high correlation bands and bands within adjacent 10nm together constitute a high correlation band set. Each rare earth element high correlation band set is input into the random forest model, and the FI index of each band is calculated. Finally, the significance results of the high FI index band are tested. According to the experiment, the significance of the first 30 important bands is set. If the test bands all show extremely significant correlation, the first 40 bands are tested, and so on (the characteristic band screening results in the actual experiment do not exceed 30). Finally, each rare earth element characteristic band is screened out.

[0125] In some embodiments, step S400 is specifically implemented by the following steps:

[0126] S401: Initial screening of high correlation bands (based on Pearson correlation coefficient).

[0127] For each band, the Pearson correlation coefficient between its spectral characteristic value and the rare earth element content was calculated.

[0128] Filtering condition: retain the absolute value of the correlation coefficient | r j The bands with ∣>0.1 were preliminarily screened out with bands significantly correlated with element content.

[0129] S402: Significance test (excluding chance).

[0130] Hypothesis testing: Perform a significance test (t-test or similar method) on the high-correlation bands in the initial screening to determine whether the correlation coefficient is significantly non-zero.

[0131] Threshold setting: Select the band with a significance level of p < 0.01 to exclude accidental correlation caused by small sample size or noise.

[0132] S403: Expand the adjacent wavelength range (cover the absorption peak region).

[0133] Extension rule: In the extremely significant correlation band (p < 0.01 and |r j |>0.1), expand 10nm to both sides:

[0134] Left boundary: the minimum value of the extremely significant correlation band minus 10nm;

[0135] Right boundary: the maximum value of the extremely significant correlation band plus 10nm.

[0136] Purpose: To avoid missing characteristic bands due to the continuous distribution of spectral absorption peaks.

[0137] S404: Random forest importance test (FI index calculation).

[0138] Model construction: The expanded set of highly correlated bands was input into the random forest regression model, with element content as the dependent variable and band spectral values ​​as the features.

[0139] Importance calculation: Calculate the importance of each band through the formula.

[0140] Sorting and screening: Sort by FI index (i.e. I(f)) from high to low, giving priority to retaining bands with high importance.

[0141] S405: Dynamic significance test and final screening.

[0142] Dynamic adjustment rules: First test the first 30 most important bands; if all pass the significance test (p < 0.01), expand the scope to the first 40 bands, and so on, until an insignificant band appears.

[0143] Output result: High correlation (|r j The bands with high significance (∣>0.1), extremely significant (p<0.01) and high importance (FI index ranking high) were identified as the characteristic bands of the rare earth elements.

[0144] S500: Construct a data set based on the characteristic bands corresponding to each element and the element chemical test content, divide the data set into a training set and a prediction set, and use the training set to train multiple machine learning models; wherein, the multiple machine learning models include a partial least squares regression model, an extreme gradient boosting machine model, a multilayer perceptron, a support vector machine regression model, and a random forest model.

[0145] This example selects multiple linear and nonlinear models for quantitative estimation of rare earth elements in sediments, including the linear model partial least squares regression (PLSR) and the nonlinear models extreme gradient boosting (XGBoost), multilayer perceptron (MLP), support vector machine regression (SVR), and random forest (RF). These methods have demonstrated good results in the estimation of terrestrial mineral resources and heavy metal elements. The estimation performance of each model is verified based on the extracted spectral and chemical data corresponding to the characteristic bands.

[0146] (1) The PLSR model has the characteristics of principal component analysis and multiple regression. The core idea is to project the predictor variables and the response variables into a new space at the same time, and find the relationship between the two through these new dimensions (called principal components, PCA), effectively representing the multiple linear relationships between the variables. It is used to simplify the model, reduce the dimension, and retain important information.

[0147] (2). The XGBoost model is an improvement of the gradient boosting algorithm (GBDT). It is one of the fastest and most effective integrated decision tree algorithms and is mainly used to deal with regression problems of large-scale and high-dimensional data sets. XGBoost generates multiple weak learners by optimizing the loss function (adding a loss function with a regularization term can effectively reduce the risk of overfitting), and each step in the learning process corrects the error of the previous step, and finally generates a powerful integrated prediction model. The XGBoost model adds regularization terms (L1 and L2 regularization) during the construction process, which effectively reduces the risk of overfitting of the model. It can also automatically handle missing values ​​and can achieve parallel computing. This embodiment uses Bayesian optimization to obtain the optimal model parameters based on 10-fold cross validation.

[0148] (3) MLP is a type of deep learning model and a powerful algorithm for dealing with nonlinear problems. It contains at least three layers of nodes: an input layer, one or more hidden layers, and an output layer, such as Figure 6 As shown in Figure 2. Each node (also called a neuron) except the input layer uses a nonlinear activation function, which is suitable for solving complex pattern recognition and prediction problems, including classification and regression tasks.

[0149] In this example, the hidden layer of the MLP model uses rectified linear units (ReLUs) and backpropagation learning is performed using the adma optimizer. L2 regularization (set to 0.001) and dropout techniques are used during model training to prevent overfitting and enhance the model's generalization ability. Spectral feature data and rare earth chemical element content data are normalized to 0-1 and used as input. Different rare earth elements have different settings for the number of hidden layers and neurons. Multiple trials and adjustments are performed to obtain the optimal parameter settings, with a maximum number of iterations set to 300. After outputting the estimated results, the data is denormalized to obtain the estimated rare earth element content.

[0150] (4). SVR is a regression application of the support vector machine model (SVM). The core idea of ​​SVR is to find a hyperplane in the high-dimensional space of the data. This hyperplane can include as many data points as possible with the smallest error. SVR introduces a concept called ε-insensitive tube. Any prediction within this tube is considered correct (that is, the error is accepted within a small threshold ε), which helps the model ignore small, unimportant errors, thereby improving the generalization ability of the model. SVR uses the same kernel method as SVM classification. SVR can find the optimal linear regression plane in this high-dimensional space. Commonly used kernel functions include linear kernel, polynomial kernel, radial basis function (RBF) kernel and sigmoid kernel.

[0151] In this embodiment, the SVR model uses a nonlinear RBF kernel function to map data into a high-dimensional space. Bayesian optimization is used to find the optimal hyperplane parameters, determine the nonlinear relationship between the spectral feature data and the rare earth elements in the sediment, and construct an element content estimation model.

[0152] (5) The RF random forest regression model is a powerful tool based on ensemble learning and is an ensemble algorithm composed of decision trees. This embodiment trains the RF model by optimizing parameters through experiments, adjusting the number of trees and minimum node size for different rare earth elements, etc., to improve the model's generalization ability and reduce overfitting. This constructs an element content estimation model.

[0153] During model construction, in order to enable model training to include more comprehensive data types and magnitudes, make the model result evaluation more comparable, and try to ensure the natural division of training sets and prediction sets and reduce human intervention, the obtained characteristic spectral data and chemical data sets were sorted and grouped sequentially according to elemental content, with 3 data sets in each group. One set in each group was randomly selected as the prediction set, and the rest were training sets, that is, the ratio of training set to prediction set was 2:1, resulting in 35 training set data and 18 prediction set data.

[0154] S600: Use multiple trained machine learning models to predict the prediction set, compare with the actual measurement values, calculate the determination coefficient, relative performance difference, root mean square error and mean absolute percentage error of the multiple trained machine learning models, evaluate the model accuracy, and select the trained machine learning model with the highest model accuracy as the quantitative estimation model. The quantitative estimation model is used to achieve hyperspectral quantification of rare earth element content in marine sediments.

[0155] In this embodiment, the coefficient of determination (R 2 ), root mean square error (RMSE), relative performance difference (RPD), and mean absolute percentage error (MAPE) are used to evaluate and verify the estimation accuracy of the model. The calculation formulas are as follows:

[0156]

[0157]

[0158] where, obs i represents the measured value of the elemental compound content, and pred i represents the predicted value of the elemental compound content. The subscript p represents the prediction set, N is the total number of prediction samples, and SD represents the standard deviation of the measured samples.

[0159] According to the R 2 evaluation result, the closer R 2 is to 1, the better the estimation result. When R 2 < 0.30, the model prediction ability is considered poor. Based on the RPD evaluation result, it can be divided into three types. When RPD > 2.00, the model prediction ability is considered good. When 1.40 < RPD < 2.00, the model prediction ability is considered relatively good. When RPD < 1.4, the model prediction ability is considered poor. For RMSE, the closer it is to 0, the better the model. Similarly, the MAPE method evaluates the prediction result according to the relative percentage error between the predicted data and the test data (true value), which is the easiest to understand and commonly used method in laboratory analysis.

[0160] This embodiment verifies and proves that in the natural environment and when sediment samples are in a near-natural state, hyperspectral technology has the ability to identify and estimate rare earth elements in sediments. Through data inspection of spectral data under different influencing factors and its subsequent feature extraction, characteristic band selection, and estimation model results.

[0161] This embodiment estimates the elemental content of rare earth elements in marine sediments based on hyperspectral data, analyzes each rare earth element and total rare earth element respectively, compares various spectral feature extraction methods and estimation models, and selects the optimal feature extraction methods and elemental content estimation models for each element and total element. The results are shown in Table 2.

[0162] Table 2. Estimation results of each rare earth element in marine surficial sediments

[0163]

[0164]

[0165] According to the analysis of Table 2, it can be seen that:

[0166] (1) Generally speaking, the TLOG-FD-MLP method model is better for the hyperspectral prediction results of light rare earth elements, and its R 2Basically greater than 0.60, RPD>1.60. For La and Ce elements, their contents are relatively high, and the obtained spectral characteristic bands are mainly distributed in the range of 380-1000nm, showing excellent prediction results. Its prediction accuracy R 2 >0.76, RPD>2.00.

[0167] (2) For medium and heavy rare earth elements, except for Gd, Tb and Dy, the estimation results of other elements are poor, and their prediction accuracy R 2 ≤0.5, RPD<1.4, indicating that there is no characteristic band for medium and heavy rare earth elements in this band or the content is small and affected by a large number of elements, resulting in unclear spectral characteristics, so the overall performance is poor.

[0168] (3) The Sc element showed a good estimation result, and MSC was able to extract its hyperspectral features well. The MLP model was the optimal estimation model, and the final estimation result R 2 =0.65, RPD =1.73. Element Y, like most medium and heavy rare earth elements, shows poor estimation results.

[0169] (4) For the total rare earth element content, the TLOG-FD method can extract the hyperspectral characteristics of the total rare earth element content in sediments. Both the SVR and MLP models can predict the total rare earth element content well. The best prediction result is R 2 is 0.73 and RPD is 1.97.

[0170] In summary, for deep-sea sediments, the spectral feature extraction method based on TLOG-FD (i.e., steps S100-S400 in the method proposed in this application) can effectively extract the spectral feature information of rare earth elements in most sediments. The MLP model can well estimate the rare earth element content and can show good prediction results for light rare earth elements. However, for medium and heavy rare earths, except for Gd, Tb and Dy, the estimation effect is generally poor.

[0171] The results show that hyperspectral data can be used to quickly, efficiently, and over a large area to estimate the rare earth element content in deep-sea sediments, demonstrating its significant application value. During the research and analysis process, the accuracy of the estimation results was primarily influenced by optimizing the quality of the hyperspectral data, extracting spectral features, and selecting the characteristic bands for each element and the total rare earth element.

[0172] Through the experimental scene setting and sample preparation, such as Figure 2 As shown in , the test process has the natural state of the test environment and the natural state of the test target. Figure 2The experimental environment shown in the figure is used, and the hyperspectral data acquired under these conditions is analyzed to verify and select valid spectral data points. The experiment uses uniform samples, with approximately 400 spectral data points obtained for each sample to ensure the correspondence between the chemical data and the hyperspectral data. After acquiring a large amount of spectral data, statistical analysis is performed on the data, with detailed error data evaluation. Anomalous data with poor stability and significant impact on the overall spectral data are removed to ensure the repeatability and reliability of the results.

[0173] Based on the above analysis method of the original data, this embodiment conducts a comparative analysis of four types of site selection schemes. Figure 7 As shown in the figure, the results show that compared with selecting all four types of spectral point data, the spectral data averaged from the three types of points ks2, ks3, and ks4 show the highest correlation with the chemical element content, especially the La element shows a high correlation, and the maximum absolute value of the correlation in the visible light band is greater than 0.6. However, if only the ks4 type points with strong stability are selected, the results are poor, that is, the correlation between the radiation brightness and the element content is poor, and the maximum absolute value of the correlation in the visible light band is basically less than 0.6. This is because the sample spectral points are too few and have a large randomness, making it difficult to represent the spectral curve of the entire sample. Figure 7 It also shows that the use of appropriate feature extraction methods can reduce the impact of sample surface roughness, looseness, etc. on the spectral curve. After analysis, it was found that the correlation between the four types of points (ks1, ks2, ks3, ks4) and three types of points (ks2, ks3, ks4) and chemical elements selected after LOG and TLOG transformation of the spectral data did not change much, but the correlation between the four types of points selected after NOR and MSC transformation was better than that of the three types of points. Considering that this embodiment is the optimal feature extraction method and estimation model for determining rare earth elements, three types of points (ks2, ks3, ks4) were selected as the final point selection results. Finally, the three types of spectral points ks2, ks3, and ks4 were selected as the input spectral information of the sample. The points selected for each sample were averaged to obtain the most representative spectral curve for each sample ( Figure 5 ).

[0174] After environmental data optimization, the correlation between hyperspectral and chemical data under all feature extraction methods has been improved, verifying the effectiveness of the proposed method. It also shows that the main reasons affecting hyperspectral data are mixed spectrum and shadow effects (ks1 points are mainly affected by mixed spectrum and shadow), and the effects of sample looseness, surface roughness, etc. can be eliminated through feature extraction. In short, the purpose of data optimization is to obtain high-precision hyperspectral data under the application environment to enhance the robustness of the model and improve the estimation accuracy. The actual seabed situation is more complex. Factors such as biological shadows, light attenuation in water, and instrument resolution can all cause the quality of the acquired hyperspectral data to vary. However, due to experimental conditions, this embodiment takes into account possible conditions such as shadows and mixed spectra from an application perspective. The hyperspectral data obtained in this environment is used for data optimization. The use of hyperspectral data for predictive modeling can improve the applicability of the model in the actual environment.

[0175] Figure 8 The figure shows the feature extraction results of the average spectrum curve obtained based on the normalization (NOR), first-order derivative (FD), multivariate scattering correction (MSC), continuum removal (CR), logarithmic first-order derivative (LOG-FD), and reciprocal logarithmic first-order derivative (TLOG-FD) feature extraction methods. There are obvious fluctuations in the absorption peak and reflection peak between 410-800nm ​​for each method. This is mainly due to the different contents of certain element compounds. Figure 9 It can be seen that there are multiple absorption peaks in the hyperspectral data under this high-resolution condition. The main absorption peaks at 410-780nm are mostly due to metal ions (Fe 2+ ,Fe 3+ ,Mn 3+ The wavelengths around 800-1000nm are primarily affected by organic matter. If there is a correlation between rare earth elements and these organic substances, content information can be obtained through indirect estimation. In addition to the main absorption peak characteristics, the high-resolution hyperspectral data of this embodiment also exhibits many medium and weak absorption peak characteristics, most of which are concentrated in the range of 410-800nm. The subtle absorption peak variations captured by this high-resolution hyperspectral data can provide a deeper understanding of the impact of rare earth elements on spectral curves in the visible light range.

[0176] The experimental results in Table 3 demonstrate that the TLOG-FD feature extraction method achieves good estimation results for total rare earth elements and most rare earth elements. Due to the skewed distribution (asymmetric data distribution) of the spectral data in this study, the model's prediction results may be significantly biased. However, the inverse logarithmic transformation can compress high values, enhance the discrimination of low values, and reduce data heteroscedasticity. Applying first-order derivative processing to the transformed data enhances weak information, improves resolution, and highlights subtle spectral variations in the hyperspectral data. This allows the PCCR-FI method to comprehensively and effectively capture the characteristic band information of each rare earth element, thereby improving the accuracy of the estimation model. Therefore, the TLOG-FD method's enhanced data discrimination and enhanced weak information contribute to the model's excellent estimation results.

[0177] Table 3. Results of selection of characteristic bands related to marine sediment elements

[0178]

[0179] This embodiment uses the PCCR-FI characteristic band screening method to effectively identify and screen the characteristic bands of each rare earth element, avoiding accidental situations such as missed selection and wrong selection of bands. Figure 9 As shown in the figure, taking La as an example, the curve represents the correlation analysis results, the bar chart represents the importance score, and the marked area is the selected high-correlation band range. As can be seen from the figure, the characteristic band distribution obtained by different spectral feature extraction methods is inconsistent. The extraction results of spectral feature bands of different elements are sensitive to the feature extraction method. Extending to the feature extraction results of the entire rare earth element, it is found that all rare earth elements are affected by the spectral feature extraction method.

[0180] After PCCR-FI screening of characteristic bands, the number of characteristic bands selected for rare earth elements is shown in Table 3. As shown in Table 3, this method reduces the number of bands to 0.8% to 4.6% of the total number of bands, eliminating a significant amount of invalid data and avoiding the over-emphasis of a band due to accidental factors or data quality issues at the end of the data, thereby improving the model's application efficiency and accuracy. Experimental analysis shows that using correlation analysis alone to obtain characteristic bands for rare earth elements results in a large number of redundant data, low model efficiency, and poor accuracy. However, using the PCCR-FI band screening method reduces the number of characteristic bands for rare earth elements in the visible light range to approximately 20, significantly improving estimation efficiency. Furthermore, the results in Table 2 demonstrate that the characteristic bands selected using the PCCR-FI method are effective for estimating rare earth element content, demonstrating that this method can accurately locate characteristic bands for rare earth elements in marine sediments and holds great promise for application. In addition, compared with the results of hyperspectral estimation of terrestrial rare earth content, this embodiment has a larger number of characteristic bands in the 400-1000nm band. This is mainly because this embodiment uses a higher-resolution hyperspectral camera, which is more capable of acquiring subtle changes between bands, better defining the depth of spectral absorption peaks affected by rare earth elements, and achieving high-precision estimation of rare earth element content in marine sediments.

[0181] The PCCR-FI characteristic band screening method proposed in this application greatly reduces the impact of redundant rare earth element bands, and fewer bands are suitable for efficient analysis in actual field applications. During the analysis process, the number of characteristic bands selected by different spectral feature extraction methods varies. Table 3 shows the number of characteristic bands for each rare earth element and the optimal estimation model for total rare earth elements. Overall, the number of characteristic bands for medium and heavy rare earth elements is less than that for light rare earth elements (except Gd and Tb), and the final estimation effect is also poor. This is because the content of medium and heavy rare earth elements is relatively low and there are few relevant bands in this wavelength range, or they are affected by the spectral characteristics of a large number of elements. It is also difficult to find the characteristic bands of these elements using various feature extraction methods.

[0182] After obtaining the best estimation model, the rare earth element content of the prediction set was estimated and its determination coefficient (R 2 ), root mean square error (RMSE), relative performance difference (RPD) of the model, and mean absolute percentage error (MAPE). During the analysis, Taylor diagrams were used to intuitively reflect the accuracy of different feature extraction methods and model estimation results for each element. The results were compared. Figure 10 As shown in the figure, the closer the prediction result is to the red solid circle on the horizontal axis, the better the estimation result is.

[0183] The Taylor plot clearly shows that for all light rare earth elements, the TLOG-FD feature extraction method and the MLP model can better estimate the rare earth element content. The parameters are evaluated using the estimation results. According to the optimal estimation results and Figure 10 Analysis shows that the TLOG-FD-MLP model generally performs better for light rare earth elements (LEEs). However, due to the sensitivity of the ionic valence state of some elements (e.g., Ce) to feature extraction methods, LOG-FD and other methods can sometimes also achieve superior estimation results. Furthermore, for some elements (e.g., Pr), different models sometimes yield similar estimation results. These discrepancies are primarily due to hyperparameter adjustments. However, some superior models are only applicable to those elements and not to others, lacking universal applicability and making them difficult to apply. The spectral characteristics of medium and heavy LEEs are sensitive to the MSC method, and most estimation results are poor. This is primarily due to the lack of characteristic bands within the applicable wavelength range of this example, or due to their low concentrations and the influence of the spectra of numerous other elements, resulting in unclear spectral characteristics. This overall performance is also poor for Y. However, some medium and heavy LEEs (e.g., Gd) perform better. Based on previous reports, this is speculated to be due to their high correlation with LLEs or other related macroelements, allowing for better estimation results using indirect estimation. For Tb, Dy, and Sc elements, the MSC-XGBoost, MSC-RF, and MSC-MLP models showed better estimation results.

[0184] This application is the first to explore the feasibility of rapid estimation of rare earth elements using hyperspectral data in the 380-1000nm band using marine sediment samples. However, according to existing research, the characteristic absorption bands of samples such as rare earth oxides and fluorides are mainly distributed in the visible light-near-infrared band, and some elements are different from those in this embodiment. This is because the rare earth sediments in this embodiment are mixed samples, and deep-sea rare earth resources have a correlation with certain mineral elements during formation. As a result, the use of 380-1000nm hyperspectral data can still obtain better estimation results, which can be used for the assessment of rare earth resources in large-scale deep-sea sediments.

[0185] Remote sensing hyperspectral estimation of terrestrial mineral resources mainly uses visible light and infrared bands, which have the advantages of in-situ, continuous, large-scale, and rapid resource acquisition. However, due to the severe attenuation of infrared light in seawater, it is impossible to obtain effective spectral information. Therefore, this example uses high-resolution hyperspectral data in the visible light and near-infrared bands of 380-1000nm, which have weak attenuation in seawater, to explore its ability to estimate rare earth elements in marine sediments. The results are shown in Table 2. The results show that the estimation results of light rare earth elements in deep-sea sediments based on hyperspectral data are R 2>0.60, RPD>1.60, the estimation results of La and Ce elements are the best, and the model prediction accuracy reaches R 2 >0.80, RPD>2.00, the estimated results of some medium and heavy rare earth elements R 2 >0.60, indicating that hyperspectral data can be effectively used to estimate the rare earth element content of different sediments. At the same time, the estimation accuracy of total rare earth element content using this hyperspectral data is R 2 The RI reached 0.73, and the RPD reached 1.97, further validating its effectiveness. Overall, this example demonstrates that hyperspectral data can effectively estimate rare earth element content within the 380-1000 nm band. This example also demonstrates the potential of hyperspectral analysis for estimating rare earth element content in deep-sea sediments. Furthermore, according to the results in Table 3, this example utilizes the PCCR-FI method to obtain a smaller number of rare earth element characteristic bands, which can reduce computational complexity during resource assessment and improve analysis efficiency, thus possessing significant application value.

[0186] In summary, this application example uses actual marine sediments as an example. By analyzing the relationship between hyperspectral data under natural light conditions and the measured rare earth element content, it preliminarily explores the possibility of using hyperspectral data to estimate the content of marine rare earth sediment resources. Using different feature extraction methods and different models to predict rare earth element content, the results show that:

[0187] (1) The method proposed in this application can better estimate the total rare earth elements, light rare earth elements and some medium rare earth elements in the ocean. 2 Basically greater than 0.60, RPD>1.60, for light rare earth elements with high content such as La and Ce, R 2 >0.76, RPD>2.00, but the estimation ability for some medium and heavy rare earth elements is poor. This is mainly because medium and heavy rare earth elements do not have characteristic bands in the 380-1000nm range or are affected by a large number of elements, resulting in unclear spectral characteristics. Moreover, most medium and heavy rare earth elements have no correlation with elements with characteristic bands in this band, making it impossible to predict the content of medium and heavy rare earth elements through direct or indirect estimation. Overall, the feature extraction method based on the logarithmic transformed first-order derivative (TLOG-FD) is the best feature extraction method for estimating rare earth element content, and the MLP model is the best estimation model for predicting rare earth elements in marine rare earth sediments. Therefore, it is concluded that the method proposed in this application can meet the requirements for assessing the content of the main components of marine rare earth mineral resources. Experiments have demonstrated the value of establishing an in situ quantitative estimation model for marine rare earth element content based on hyperspectral technology. Furthermore, in situ detection of marine rare earth elements based on hyperspectral technology provides a new technical path for the detection and assessment of marine mineral resources.

[0188] (2) The embodiment of the present application takes into account the attenuation problem of actual seabed light and performs quantitative estimation of rare earth elements based on hyperspectral data in the 380-1000nm band, which can achieve rapid and large-scale resource assessment. The scope of application is not limited to the resource assessment of marine rare earth sediments. Hyperspectral technology can be used to conduct large-scale resource assessment of various valuable seabed mineral resources. With the advancement of technology in the future, it is expected to become an important means of marine mineral resource assessment.

[0189] The embodiments of this application are experimental studies on marine sediment samples, which demonstrate the feasibility of using hyperspectral technology to efficiently collect rare earth spectral information and quickly and widely estimate the content of rare earth elements, providing a basis for further applications.

[0190] The present application also provides a device for quantitatively estimating the content of rare earth elements. Figure 11 As shown, the device for quantitatively estimating the content of rare earth elements comprises:

[0191] The data acquisition module 1101 is configured to acquire hyperspectral data of each sample; wherein the hyperspectral data includes a plurality of data points, and each data point includes a plurality of spectral band data;

[0192] The data preprocessing module 1102 is configured to perform data dimension reduction based on the hyperspectral data of each sample by removing large error points through quartile analysis, perform K-means cluster analysis on the reduced data features to obtain valid spectral data of each sample, and perform denoising on the valid spectral data of each sample and average the data to obtain an average spectral curve;

[0193] The spectral feature extraction module 1103 is configured to extract spectral features based on the average spectral curve; wherein the spectral features include characteristic data of each band;

[0194] The characteristic band screening module 1104 is configured to use Pearson correlation to analyze the correlation between the characteristic data of each band and the element content, select a highly correlated band range, and use random forest to perform a significance test on the highly correlated band range to obtain important characteristic bands, thereby screening the characteristic bands corresponding to each element;

[0195] The model training module 1105 is configured to construct a data set based on the characteristic bands corresponding to each element and the element's chemical test content, divide the data set into a training set and a prediction set, and use the training set to train multiple machine learning models; wherein the multiple machine learning models include a partial least squares regression model, an extreme gradient boosting model, a multilayer perceptron, a support vector machine regression model, and a random forest model;

[0196] The model evaluation module 1106 is configured to use the trained multiple machine learning models to predict the prediction set, compare the actual measurement values, calculate the determination coefficient, relative performance difference, root mean square error and mean absolute percentage error of the trained multiple machine learning models, evaluate the model accuracy, and select the trained machine learning model with the highest model accuracy as the quantitative estimation model, which is used to achieve hyperspectral quantification of rare earth element content in marine sediments.

[0197] An embodiment of the present application provides an electronic device, which may include a processor and a memory, wherein the processor and the memory can communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0198] The processor executes the computer-executable instructions stored in the memory, so that the processor implements the solutions in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0199] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. System buses can be categorized as address buses, data buses, and control buses. Transceivers enable communication between the database access device and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) or non-volatile memory.

[0200] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0201] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solution of the method for quantitative estimation of rare earth element content in the above embodiment.

[0202] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the method for quantitative estimation of rare earth element content in the above embodiment.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0204] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0205] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The above-mentioned modules may be implemented in the form of hardware or hardware plus software functional units.

[0206] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0207] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0208] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0209] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, and control buses.

[0210] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0211] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0212] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for quantitatively estimating rare earth element content, characterized in that: The method comprises: Acquire hyperspectral data of each sample; wherein the hyperspectral data includes a plurality of data points, and each data point includes a plurality of spectral band data; Based on the hyperspectral data of each sample, large error points are eliminated by quartiles to perform data dimensionality reduction, and the data features after dimensionality reduction are subjected to K-means cluster analysis to obtain the effective spectral data of each sample, and the effective spectral data of each sample are denoised and averaged to obtain an average spectral curve; Extracting spectral features based on the average spectral curve; wherein the spectral features include characteristic data of each band; The correlation between the characteristic data of each band and the element content was analyzed using Pearson correlation, and the highly correlated band range was selected. The importance of the highly correlated band range was tested using random forest to obtain important characteristic bands and screen out the characteristic bands corresponding to each element. A data set is constructed by combining the characteristic bands corresponding to each element and the element's chemical test content, and the data set is divided into a training set and a prediction set. The training set is used to train multiple machine learning models; wherein the multiple machine learning models include a partial least squares regression model, an extreme gradient boosting model, a multilayer perceptron, a support vector machine regression model, and a random forest model; The prediction set was predicted using multiple trained machine learning models, and compared with the actual measurement values. The determination coefficients, relative performance differences, root mean square errors, and mean absolute percentage errors of the trained machine learning models were calculated to evaluate the model accuracy. The trained machine learning model with the highest model accuracy was selected as the quantitative estimation model, which was used to achieve hyperspectral quantification of rare earth element content in marine sediments.

2. The method for quantitatively estimating rare earth element content according to claim 1, wherein: Based on the hyperspectral data of each sample, large error points are eliminated by quartiles, and the data dimension reduction method includes: The acquired hyperspectral data are synthesized into a matrix and standardized using the following formula: Where, σ i is the standard deviation of the i-th sample, X i is the random variable of the i-th sample, is the sample mean of the i-th sample, x i is the feature of the i-th sample in the standardized sample matrix; The covariance matrix R of the standardized sample matrix is ​​calculated as follows: Where, X ′ is the standardized data matrix, T is the matrix transpose, and N is the number of samples; Calculate the eigenvalues ​​and eigenvectors of the covariance matrix R, where the eigenvectors represent the directions of the principal components and the magnitude of the eigenvalues ​​represents the importance of each principal component. Select the first m eigenvectors with the largest eigenvalues ​​as the principal components to obtain the data features after dimensionality reduction.

3. The method for quantitatively estimating rare earth element content according to claim 1, wherein: The methods for performing K-means cluster analysis on the data features after dimensionality reduction to obtain the effective spectral data of each sample include: Randomly select K data points as initial cluster centers; Assignment: Calculate the distance between each data point and each cluster center, and assign the data point to the cluster center closest to it; Update: For each cluster center, after adding a new data point, the mean of all data points in the cluster is used as the new cluster center; The allocation and updating are repeated until a set condition is met, wherein the set condition includes reaching a set number of iterations.

4. The method for quantitatively estimating rare earth element content according to claim 1, wherein: Based on the average spectral curve, the spectral features are extracted by normalization, first-order derivative, multivariate scattering correction, continuum removal, logarithmic first-order derivative and reciprocal logarithmic first-order derivative.

5. The method for quantitatively estimating rare earth element content according to claim 1, wherein: The Pearson correlation is used to analyze the correlation between the characteristic data of each band and the element content. The methods for selecting the high correlation band range include: The correlation coefficient is calculated using the following formula: Where r j is the correlation coefficient of the jth band, x i Indicates the content of an element, y i represents the spectral characteristic value under the j-th band, and Indicates the sample mean of the two sets of data analyzed in this band; Setting a correlation coefficient threshold, and determining a band with a correlation coefficient greater than the correlation coefficient threshold as a high-correlation band; The maximum value of all high-correlation bands is increased by 10 nm as the rightmost band range, and the minimum value is reduced by 10 nm as the leftmost band range to obtain the high-correlation band range.

6. The method for quantitatively estimating rare earth element content according to claim 5, characterized in that: Methods for using random forest to perform importance test on the highly correlated band range to obtain important characteristic bands and screen out characteristic bands corresponding to each element include: Inputting the highly correlated band range into a random forest regression model, wherein the random forest regression model takes element content as a dependent variable and band spectrum value as a feature; The importance of each band is calculated using the following formula: Where, ΔImpurity n1 (f) is the purity reduction value of the feature band f at node n1, T1 is the total number of trees, I(f) represents the importance of the band, which is the average impurity reduction of the feature band f in all trees, t is the ordinal number of the tree, and n1 is the node of the tree; According to the importance of each band, the characteristic bands corresponding to each element are screened out.

7. The method for quantitatively estimating rare earth element content according to claim 6, characterized in that: According to the importance of each band, the characteristic bands corresponding to each element are screened out, including: Test the first m most important bands; if all pass the significance test, expand the range to the first m + m1 bands, and so on, until an insignificant band appears, and obtain the characteristic band corresponding to each element; where m and m1 are set constants.

8. A device for quantitatively estimating rare earth element content, characterized in that: The device comprises: A data acquisition module is configured to acquire hyperspectral data of each sample; wherein the hyperspectral data includes a plurality of data points, and each data point includes a plurality of spectral band data; The data preprocessing module is configured to perform data dimension reduction based on the hyperspectral data of each sample by eliminating large error points through quartiles, perform K-means cluster analysis on the data features after dimension reduction to obtain effective spectral data of each sample, and perform denoising on the effective spectral data of each sample and average the data to obtain an average spectral curve; A spectral feature extraction module is configured to extract spectral features based on the average spectral curve; wherein the spectral features include characteristic data of each band; The characteristic band screening module is configured to use Pearson correlation to analyze the correlation between the characteristic data of each band and the element content, select a highly correlated band range, and use random forest to perform an importance test on the highly correlated band range to obtain important characteristic bands, and screen out the characteristic bands corresponding to each element; A model training module is configured to construct a data set based on the characteristic bands corresponding to each element and the element's chemical test content, divide the data set into a training set and a prediction set, and use the training set to train multiple machine learning models; wherein the multiple machine learning models include a partial least squares regression model, an extreme gradient boosting machine model, a multilayer perceptron, a support vector machine regression model, and a random forest model; The model evaluation module is configured to use multiple trained machine learning models to predict the prediction set, compare the actual measurement values, calculate the determination coefficient, relative performance difference, root mean square error and mean absolute percentage error of the multiple trained machine learning models, evaluate the model accuracy, and select the trained machine learning model with the highest model accuracy as the quantitative estimation model, which is used to achieve hyperspectral quantification of rare earth element content in marine sediments.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for quantitatively estimating the rare earth element content according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for quantitatively estimating the content of rare earth elements according to any one of claims 1 to 7.

Images