Field rock ore spectral data preprocessing method
Through the integration of data preprocessing, feature extraction and DBSCAN clustering analysis methods, the noise and complexity problems in field spectral measurement data are solved, the data processing efficiency and accuracy are improved, and it is suitable for research and application in multiple fields.
Patent Information
- Application Number
- CN202410996249.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
There is excessive noise and complexity in field spectral measurement data, resulting in low data processing efficiency and poor accuracy, and it is difficult for the prior art to eliminate noise quickly and effectively.
Adopting integrated data preprocessing, feature extraction, dimensionality reduction and density-based clustering analysis methods, the DBSCAN algorithm is used to identify and process noise, reduce data complexity through automated steps, retain key information, and provide data visualization tools to improve analysis transparency.
It improves data processing efficiency and accuracy, optimizes data analysis quality, effectively processes complex data structures, enhances the interpretability and visualization capabilities of spectral data, and is suitable for research in field spectral data and other fields.
Smart Images

Figure CN118940070A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of noise removal of ground object spectral data and relates to a field rock and mineral spectral data preprocessing method. Background Art
[0002] The geological and mineral industry is one of the earliest and most important application areas of spectral testing and analysis. Through the analysis of the spectral characteristics of rocks and minerals, various rocks, minerals and their various physical and chemical characteristics can be identified, providing a new means for the theoretical research of rocks and minerals. In specific geological and mineral surveys or scientific research projects, rock and mineral samples are measured in the field or indoors by using a ground object spectrometer, and reflectance data can be obtained after processing. By matching the spectral data with the standard library data, not only can the mineral type be identified, but the mineral composition can also be inverted based on the spectral characteristics.
[0003] Field surface spectral measurement is to conduct on-site spectral measurement in the field work area. It can quickly obtain the spectral emission characteristics of rocks and minerals without sample collection, processing and test analysis. Field spectral measurement can be of great significance for aerospace sensor calibration, remote sensing data interpretation, and remote sensing application potential research. In the field of hyperspectral remote sensing, it has gradually developed from the laboratory research stage to the practical application stage, and is used as an important technical means in mineral resource surveys. A large number of surface spectral observations are required to study the visible and near-infrared band reflectance spectra of various different surface objects or environments under natural conditions in the field and to master the spectral radiation characteristics of various surface objects on the ground. Field surface spectral measurement is a complex process that requires comprehensive consideration of various spectral influencing factors. The spectral data obtained is the result of the combined effects of various factors such as solar altitude angle, solar azimuth angle, cloud, wind, relative humidity, incident angle, detection angle, instrument scanning speed, instrument field of view angle, instrument sampling interval, spectral resolution, slope aspect, slope and spectral characteristics of the target itself. Field spectral measurements generally collect data for known target minerals or rocks. Due to the complexity of field conditions, it is inevitable that there are some invalid or noisy data in the measured data. This also includes the deviation of field identification of rock and mineral samples, resulting in the actual measured samples not being the recorded type. This can be eliminated through analysis with relevant software such as ENVI, but how to quickly eliminate noise from a large amount of data still needs further discussion. Summary of the invention
[0004] In the field rock and mineral spectrum collection, the collected data are generally several samples of known types, and the spectra of samples of the same type have similar characteristics. The spectral characteristics of the noise or invalid data have a large deviation from the valid data. Based on this, the present invention provides a field rock and mineral spectrum data preprocessing method for the existing technical problems.
[0005] A method for preprocessing field rock and mineral spectral data comprises the following steps: an integrated data preprocessing step, a data screening and feature extraction step, a cluster analysis step, and a data visualization and analysis step, which identifies and extracts the most informative features in the data, reduces the complexity of the data, retains the key information of the data, uses the DBSCAN algorithm to perform cluster analysis, identifies and processes the noise in the data, finds data clusters with arbitrary shapes, obtains data visualization through precise data preprocessing, effective feature extraction and dimensionality reduction, and sophisticated cluster analysis, and optimizes the analysis process of spectral data.
[0006] The advantage of the present invention is that it provides a method for efficiently processing and analyzing a large amount of field measured spectral data based on integrated data preprocessing, feature extraction, dimensionality reduction, and density-based cluster analysis steps. This method is not only suitable for the analysis of field spectral data, but can also be extended to multiple fields such as environmental monitoring, geological exploration, and agricultural research. Its efficient and reliable data processing capabilities provide a powerful tool for research and application in related fields and have significant practical value. The specific technical effects are as follows:
[0007] 1. Improve data processing efficiency and accuracy:
[0008] Through automated data preprocessing steps, the system can automatically identify the starting rows of valid data and exclude non-data rows, reducing human intervention and thus the possibility of human error; the data reading and organization steps ensure the structured storage of data, providing a clear data framework for subsequent analysis, significantly improving processing speed and accuracy, especially when processing large-scale field measured spectral data sets.
[0009] 2. Optimize data analysis quality:
[0010] The feature extraction and dimensionality reduction steps ensure the extraction of the most informative features in the data by screening specific wavelength regions and normalizing the data. Principal component analysis (PCA) further reduces the complexity of the data while retaining the most important variation information in the data; this optimization not only improves the quality and reliability of data analysis, but also makes subsequent clustering analysis more accurate and meaningful.
[0011] 3. Effectively handle complex data structures:
[0012] Using the DBSCAN algorithm for cluster analysis, the system can effectively identify and process the noise in the data and find data clusters with arbitrary shapes. The advantage of the DBSCAN algorithm is that it can process nonlinear and complex distributed data and reveal the complex structures and patterns hidden in the data; the parameter estimation step accurately estimates the ε value and MinPts parameter of the DBSCAN algorithm through the k-distance graph to ensure the resolution and accuracy of the clustering.
[0013] 4. Enhanced spectral data interpretation and visualization capabilities:
[0014] The data visualization tools provided, such as the original spectral data graph and k-distance graph, enable researchers to intuitively understand the data characteristics and clustering results. This visualization support enhances the transparency and understanding of data analysis, helps to deeply analyze and interpret the data, and improves data interpretability.
[0015] 5. Application potential in multiple fields:
[0016] This method is not only suitable for the analysis of field spectral data, but can also be extended to many fields such as environmental monitoring, geological exploration, agricultural research, etc. Its efficient and reliable data processing capabilities provide a practical tool for research and application in related fields.
[0017] 6. Good practical application effect:
[0018] The significant effect of this method in processing field measured spectral data is demonstrated through specific examples. The comprehensive application of steps such as automatic identification of the starting row of valid data, screening of specific wavelength regions, data normalization, principal component analysis and DBSCAN clustering ensures the purity of the data, reduces complexity and effectively removes noise; the DBSCAN parameters are estimated through k-distance graphs, demonstrating the transparency and understanding of parameter selection, and the cluster analysis results intuitively demonstrate the natural clustering and noise distinction effect in the data. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. As shown in the figure:
[0020] Figure 1 It is a flow chart of the present invention.
[0021] Figure 2 This is the original spectrum data diagram of the present invention.
[0022] Figure 3 This is a diagram for automatically acquiring the core parameters of the present invention.
[0023] Figure 4 This is the automatic noise removal result of the present invention.
[0024] Figure 5 This is a data text diagram after format conversion according to the present invention. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] Example 1: Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, a method for preprocessing field rock and mineral spectral data is provided, which automatically and quickly removes noise or mismeasured data from a large amount of spectral data collected in the field in response to the specific needs of field measured spectral data processing and analysis.
[0027] A comprehensive approach to optimize the analysis of spectral data through precise data preprocessing, effective feature extraction and dimensionality reduction, sophisticated clustering analysis, and intuitive data visualization. It provides a set of targeted solutions to common challenges in large-scale spectral datasets, such as noise processing, high data dimensionality, and complex data structure.
[0028] A method for preprocessing field rock and mineral spectral data comprises the following steps: an integrated data preprocessing step, a data screening and feature extraction step, a cluster analysis step and a data visualization and analysis step, which identifies and extracts the most informative features in the data, reduces the complexity of the data, retains the key information of the data, uses the DBSCAN algorithm to perform cluster analysis, identifies and processes the noise in the data, finds data clusters with arbitrary shapes, obtains data visualization through precise data preprocessing, effective feature extraction and dimensionality reduction, and sophisticated cluster analysis, and optimizes the analysis process of spectral data.
[0029] The integrated data preprocessing step includes the following steps:
[0030] Automatically identify valid data start lines: By traversing each line of the file, automatically distinguish and exclude non-data lines containing text (titles, comments), and only retain lines containing numerical data.
[0031] Data reading and organization: Read the wavelengths and corresponding measurement values as structured data to provide a clear data framework for subsequent analysis.
[0032] The data screening and feature extraction steps include the following steps:
[0033] Filter specific wavelength regions: Focus on data with wavelengths greater than 700nm, and based on the principles of spectroscopy, filter out spectral regions that are more representative of specific research.
[0034] Data normalization: Normalize the filtered data to eliminate the dimension effect and enhance the comparability between different samples.
[0035] Principal Component Analysis (PCA): PCA is used to reduce the dimension of normalized data, extract the main components, reduce data complexity, and retain the most important variation information in the data.
[0036] The cluster analysis steps include the following steps:
[0037] Parameter estimation (ε and MinPts): The k-distance graph is used to estimate the ε value of the DBSCAN algorithm and determine the optimal neighborhood size. The MinPts parameter is set according to the number of components after PCA dimensionality reduction, aiming to balance the density of the data and the resolution of the clustering.
[0038] DBSCAN clustering: Using the estimated ε and MinPts as parameters, perform DBSCAN clustering to identify natural clusters in the data and distinguish noise.
[0039] The data visualization and analysis steps include the following steps:
[0040] Original spectral data graph: Displays the original spectral data in a visual way, providing an intuitive data overview.
[0041] k-distance plot: Plot a k-distance plot and annotate the estimated ε values to visually demonstrate the process of determining DBSCAN parameters, enhancing the transparency and understanding of parameter selection.
[0042] Example 2: Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown in FIG. 1 , a method for preprocessing field rock and mineral spectral data mainly includes:
[0043] 1. Data preprocessing
[0044] (1) Automatically identify the starting row of valid data:
[0045] By traversing each line of the file, using regular expressions or other string matching methods, it automatically distinguishes and excludes non-data lines containing text (such as titles, comments), and only retains lines containing numerical data. This step ensures the purity and accuracy of the data in subsequent processing steps.
[0046] (2) Data reading and organization:
[0047] Read the wavelength and the corresponding measured value as structured data to provide a clear data framework. Assume that the data contains n rows, each containing a wavelength value λ iand the corresponding measured value y i , then the data set can be expressed as:
[0048] {(λ1,y1O,(λ2,y2O,…,(λ n ,y n )}
[0049] where λ i is the wavelength of the ith data point, y i is the measured value of the ith data point.
[0050] 2. Data screening and feature extraction
[0051] (1) Screening for specific wavelength regions:
[0052] We focus on data with wavelengths greater than 700nm and filter out spectral regions that are more representative of specific studies based on the principles of spectroscopy. The filtered data sets are:
[0053] {(λ i ,y i )∣λ i >700nm}
[0054] (2) Data normalization:
[0055] The screened data are normalized to eliminate the impact of dimension and enhance the comparability between different samples.
[0056] (3) Principal Component Analysis (PCA)
[0057] PCA is used to reduce the dimension of normalized data and extract the main components, which significantly reduces the complexity of the data while retaining the most important variation information in the data. The specific PCA steps are as follows:
[0058] 1) Standardized data:
[0059] X=[x1,x2,…,x n ]
[0060] in x i The standardized data, y i is the original data, μ is the mean of the data, and σ is the standard deviation of the data.
[0061] 2) Calculate the covariance matrix:
[0062]
[0063] Sample mean vector where C is the covariance matrix, is the sample mean vector and n is the number of data points.
[0064] 3) Calculate the eigenvalues and eigenvectors of the covariance matrix:
[0065] C(=λ(
[0066] Among them, λ is the eigenvalue and v is the eigenvector.
[0067] 4) Select the principal component
[0068] Select the eigenvectors corresponding to the first k eigenvalues in descending order to form a new eigenvector matrix) * . Project the data into the new feature space:
[0069] + = X) *
[0070] Where + is the data matrix after projection,) * is the matrix composed of the first eigenvectors.
[0071] 3. Cluster analysis
[0072] The main purpose of cluster analysis is to identify natural clusters in the data and distinguish noise points. We use DBSCAN (density-based clustering algorithm) to accomplish this task.
[0073] (1) Parameter estimation
[0074] Before applying the DBSCAN algorithm, two key parameters must be determined: ε (epsilon, neighborhood radius) and MinPts (minimum number of points). Reasonable parameter selection is crucial to the quality of clustering results.
[0075] 1) Calculate k-distance:
[0076] By calculating the distance of each data point to its kth nearest neighbor, we can help determine the value of ε.
[0077] 2) Draw the k-distance graph:
[0078] Calculate the distance from each data point to its kth nearest neighbor, then sort them in ascending order and plot the k-distance graph. The inflection point in the graph usually corresponds to a suitable ε value.
[0079] 3) Determine MinPts:
[0080] The MinPts parameter can be determined based on domain knowledge or through experience. In general, MinPts should be at least equal to the dimension of the dataset (k) plus 1.
[0081] (2) DBSCAN clustering:
[0082] After determining the ε and MinPts parameters, perform DBSCAN clustering:
[0083] 1) Define the ε neighborhood:
[0084] - ε (.(= / 0∈2|3i45).,0)≤ε7
[0085] Among them: -8(.) is the 9-domain of point ., 2 is the data set, and 3i45(.,0) is the distance between point . and 0.
[0086] 2) Core conditions:
[0087] |- ε (.)|≥MinPts
[0088] 3) Clustering process:
[0089] For each unvisited data point, if it is a core point, create a new cluster from the beginning and expand all reachable points. Repeat this process until all points have been visited.
[0090] 4. Data Visualization and Analysis
[0091] (1) Original spectrum data:
[0092] The raw spectral data is presented visually, showing the relationship between wavelength and reflectance, providing researchers with an intuitive data overview.
[0093] (2) k-distance graph:
[0094] Plot the k-distance graph and annotate the estimated ε value to show the process of determining DBSCAN parameters, enhancing the transparency and understanding of parameter selection.
[0095] Through the above-mentioned detailed technical solutions and formula derivation, the scientificity and effectiveness of this method in processing a large amount of field measured spectral data are demonstrated. Automated data preprocessing reduces human errors and ensures data accuracy. Through screening, normalization and PCA dimensionality reduction, the most valuable information in the data is effectively extracted, reducing the complexity of data processing. DBSCAN cluster analysis further demonstrates the significant features of complex data structures and enhances data interpretability and visualization capabilities.
[0096] Example 3: Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, a method for preprocessing field rock and mineral spectral data, with 114 field measured spectral data as test data, includes the following steps:
[0097] Step 1: Convert the ground object spectrometer measurement data into ASCII-encoded txt data. The result is Figure 5 As shown:
[0098] In matlab, read the data and save it as a multidimensional array (114 dimensions) and display it as follows Figure 2 As shown in the figure: In DBSCAN clustering, the elbow rule is adaptively used to determine the value of ε (epsilon, i.e., neighborhood radius) rather than directly used to determine the number of clusters. This is because the working mechanism of the DBSCAN algorithm is based on density, and the ε value is used to determine whether the data points belong to the same cluster. Therefore, determining a suitable ε value is crucial to the effectiveness of the DBSCAN algorithm.
[0099] Use the k-distance graph to determine the epsilon parameter of the DBSCAN algorithm: calculate the Euclidean distance between all points, find the distance of each point to its MinPtsth nearest neighbor, and estimate the optimal epsilon value by drawing the k-distance graph and finding the inflection point of the graph. The processing results of the surface data are as follows Figure 3 Shown: Plot a k-distance plot and mark the inflection points: Displays a k-distance plot where the inflection points are used to estimate the epsilon parameter for the DBSCAN clustering algorithm.
[0100] Select the first 10 components of PCA as the main components, and the automatic noise removal results are as follows Figure 4 As shown, it can be seen that the detailed noise has been successfully removed.
[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for preprocessing field rock and mineral spectral data, characterized in that: The method includes the following steps: integrated data preprocessing step, data screening and feature extraction step, cluster analysis step and data visualization and analysis step, which can identify and extract the most informative features in the data, reduce the complexity of the data, retain the key information of the data, use the DBSCAN algorithm for cluster analysis, identify and process the noise in the data, find data clusters with arbitrary shapes, obtain data visualization through precise data preprocessing, effective feature extraction and dimensionality reduction, and sophisticated cluster analysis, and optimize the analysis process of spectral data.
2. A method for preprocessing field rock and mineral spectral data according to claim 1, characterized in that: The integrated data preprocessing step includes the following steps: Automatically identify the starting line of valid data: By traversing each line of the file, it automatically distinguishes and excludes non-data lines containing text (titles, comments), and only retains lines containing numerical data. Data reading and organization: Read the wavelengths and corresponding measurement values as structured data to provide a clear data framework for subsequent analysis.
3. A method for preprocessing field rock and mineral spectral data according to claim 1, characterized in that: The data screening and feature extraction steps include the following steps: Filter specific wavelength regions: Focus on data with wavelengths greater than 700nm, and based on the principles of spectroscopy, filter out spectral regions that are more representative of specific research. Data normalization: Normalize the filtered data to eliminate the dimension effect and enhance the comparability between different samples. Principal Component Analysis (PCA): PCA is used to reduce the dimension of normalized data, extract the main components, reduce data complexity, and retain the most important variation information in the data.
4. A method for preprocessing field rock and mineral spectral data according to claim 1, characterized in that: The cluster analysis steps include the following steps: Parameter estimation: The k-distance graph is used to estimate the ε (epsilon, community radius) value of the DBSCAN algorithm to determine the optimal neighborhood size. The MinPts (minimum number of points) parameter is set according to the number of components after PCA dimensionality reduction, aiming to balance the density of the data and the resolution of the clustering. DBSCAN clustering: Using the estimated ε and MinPts as parameters, perform DBSCAN clustering to identify natural clusters in the data and distinguish noise.
5. A method for preprocessing field rock and mineral spectral data according to claim 1, characterized in that: The data visualization and analysis steps include the following steps: Original spectral data graph: Displays the original spectral data in a visual way, providing an intuitive data overview. k-distance plot: Plot a k-distance plot and annotate the estimated ε values to visually demonstrate the process of determining DBSCAN parameters, enhancing the transparency and understanding of parameter selection.
6. A method for preprocessing field rock and mineral spectral data according to claim 2, characterized in that: The data visualization and analysis steps include the following steps: (1) Automatically identify the starting row of valid data: By traversing each line of the file, using regular expressions or other string matching methods, it automatically distinguishes and excludes non-data lines containing text (titles, comments), and only retains lines containing numerical data. This step ensures the purity and accuracy of the data in subsequent processing steps. (2) Data reading and organization: Read the wavelength and the corresponding measured value as structured data to provide a clear data framework. Assume that the data contains n rows, each containing a wavelength value λ i and the corresponding measured value y i , then the data set can be expressed as: {(λ1,y1),(λ2,y2),…,(λ n ,y n )}。 7. A method for preprocessing field rock and mineral spectrum data according to claim 3, characterized in that: The data screening and feature extraction steps include the following steps: (1) Screening specific wavelength regions: Focusing on data with wavelengths greater than 700nm, based on the principles of spectroscopy, we screened out spectral regions that are more representative of specific studies. The screened data sets are: {(l i ,y i )∣λ i >700nm} where λ i is the wavelength of the ith data point, y i is the measured value of the ith data point, (2) Data normalization: Normalize the filtered data to eliminate the dimension effect and enhance the comparability between different samples. (3) Principal Component Analysis (PCA) PCA is used to reduce the dimension of normalized data, extract the main components, significantly reduce the complexity of the data, and retain the most important variation information in the data. The specific PCA steps are as follows: 1) Standardized data: X=[x1,x2,…,x n ] in x i The standardized data, y i is the original data, μ is the mean of the data, σ is the standard deviation of the data, 2) Calculate the covariance matrix: Sample mean vector where C is the covariance matrix, is the sample mean vector, n is the number of data points, 3) Calculate the eigenvalues and eigenvectors of the covariance matrix: Cv=λv Among them, λ is the eigenvalue, v is the eigenvector, 4) Select the principal component Select the eigenvectors corresponding to the first k eigenvalues in descending order to form a new eigenvector matrix V * , the data is projected into the new feature space: Z=XV * Where Z is the projected data matrix, V * is the matrix consisting of the first k eigenvectors.
8. A method for preprocessing field rock and mineral spectrum data according to claim 3, characterized in that: Cluster analysis steps, including the following steps: The main purpose of cluster analysis is to identify natural clusters in the data and distinguish noise points. DBSCAN (density-based clustering algorithm) is used to complete this task. (1) Parameter estimation Before applying the DBSCAN algorithm, two key parameters must be determined: ε (epsilon, neighborhood radius) and MinPts (minimum number of points). Reasonable parameter selection is crucial to the quality of clustering results. 1) Calculate k-distance: By calculating the distance of each data point to its kth nearest neighbor, we can help determine the value of ε. 2) Draw the k-distance graph: Calculate the distance from each data point to its kth nearest neighbor, then sort them in ascending order and draw a k-distance graph. The inflection point in the graph usually corresponds to a suitable ε value. 3) Determine MinPts: The MinPts parameter can be determined based on domain knowledge or through experience. MinPts should be at least equal to the dimension of the dataset (k) plus 1. (2) DBSCAN clustering: After determining the ε and MinPts parameters, perform DBSCAN clustering: 1) Define the ε neighborhood: N ε (p)= / q∈D∣dist(p,q)≤ε7 Where: N8(p) is the ∈ domain of point p, D is the data set, dist(p,q) is the distance between points q and p, 2) Core conditions: |N ε (p)|≥MinPts 3) Clustering process: For each unvisited data point, if it is a core point, create a new cluster from the beginning, expand all reachable points, and repeat this process until all points have been visited.
9. A method for preprocessing field rock and mineral spectrum data according to claim 4, characterized in that: The data visualization and analysis steps include the following steps: (1) Original spectrum data: By visualizing the raw spectral data and showing the relationship between wavelength and reflectance, it provides researchers with an intuitive data overview. (2) k-distance graph: Plot the k-distance graph and annotate the estimated ε value to show the process of determining DBSCAN parameters, enhancing the transparency and understanding of parameter selection.
Citation Information
Patent Citations
Novel rough-set-based dimension reduction method for hyperspectral remote sensing data
CN106971204A
A wavelength selection method based on PCA modeling feedback load weighting
CN109409350A
Spectrum classification method based on PCA-UVE-ELM
CN113408616A
Abnormality detection method for SERS spectral data of serum
CN117056840A
ReaxFF post-processing method for extracting product, cluster and chemical bond information based on Python
CN117423394A
Cited By
Method and device for automatically removing scattering noise of 3D fluorescence spectrum and readable storage medium
CN120336724A
Method, device and readable storage medium for automatic removal of 3D fluorescence spectrum scattering noise
CN120336724B
Spectral data processing method and spectral data processing device
CN121167226A
Spectral data processing method and spectral data processing apparatus
CN121167226B
