Ground measurement spectrum data quality auditing method based on reference spectrum library

Abstract

The invention belongs to the technical field of spectral measurement, relates to a ground measurement spectral data quality auditing method based on a reference spectrum library, and solves the problem of improving the efficiency and accuracy of data quality auditing. The method comprises the steps of establishing a reference spectrum library, preprocessing measured spectrum data, calculating similarity, detecting abnormal points and generating a quality auditing report. According to the invention, the measurement spectrum is compared with the pre-established reference spectrum library, the abnormal points in the data are automatically identified and marked, and the efficiency and accuracy of data quality auditing are improved. According to the ground measurement spectrum data quality auditing method based on the reference spectrum library, a detailed quality auditing report is generated through establishment of the reference spectrum library, preprocessing of measurement spectrum data, similarity calculation of a measurement spectrum and a reference spectrum, abnormal point detection and spectrum quality evaluation. The method is suitable for spectral data processing in the fields of environmental monitoring, agricultural remote sensing, geological exploration and the like, and has a wide application prospect.

Description

Technical Field

[0001] The invention belongs to the technical field of spectral measurement, and in particular relates to a method for auditing the quality of ground measurement spectral data based on a reference spectral library. Background Art

[0002] In the fields of environmental monitoring, agricultural remote sensing, geological exploration, etc., the accuracy and reliability of ground-measured spectral data are of great significance for subsequent data analysis and decision-making. However, ground-measured spectral data may be affected by many factors, such as atmospheric conditions, instrument noise, and measurement errors, resulting in uneven data quality. Traditional manual quality review methods are inefficient and error-prone, and cannot meet the needs of large-scale data processing. Therefore, it is particularly important to develop an automated and efficient spectral data quality review method. Summary of the invention

[0003] The purpose of the present invention is to provide a method for quality auditing of ground measurement spectral data based on a reference spectral library, which solves the problem of improving the efficiency and accuracy of data quality auditing through automated similarity calculation and outlier detection. This method can be used for automatic quality auditing of ground measurement spectral data.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A method for reviewing the quality of ground-based spectral data based on a reference spectral library is to automatically identify and mark abnormal points in the data by comparing the measured spectrum with a pre-established reference spectral library, including the following steps:

[0006] A. Establish a reference spectrum library: collect and organize the measured spectrum data of various standard samples, and establish a reference spectrum library containing a variety of typical spectral features; organize the spectra of clear and cloudless weather measured at field sites, remove invalid spectra, and merge similar spectra; obtain a reference spectrum library containing various typical landforms;

[0007] B. Preprocessing of measured spectral data: De-noising, smoothing and baseline correction preprocessing operations are performed on the spectral data obtained from ground measurements to eliminate instrument noise and other interferences;

[0008] C. Similarity calculation: Using similarity measurement methods such as cosine similarity or Euclidean distance, the preprocessed measured spectrum is compared with the spectrum in the reference spectrum library to calculate the similarity between the two;

[0009] D. Outlier detection: Set a similarity threshold. When the similarity between the measured spectrum and the reference spectrum is lower than the threshold, it is determined as an outlier and marked.

[0010] E. Generate quality audit report: Based on the anomaly marking, generate a detailed quality audit report indicating data quality issues and their locations.

[0011] Furthermore, step A specifically includes the following steps:

[0012] A1. Data collection: Use the spectra of standard samples measured and sorted at ground field stations as the input spectral data;

[0013] A2. Data collation: Standardize the collected spectral data to ensure uniform data format and wavelength range, and eliminate spectra with incorrect format or incomplete content;

[0014] A3. Build a reference spectral library: Add the sorted spectral data to the reference spectral library according to the dynamic clustering method to build the optimal spectral library.

[0015] Furthermore, in step A3, the calculation process of building the reference spectral library is as follows:

[0016] A31. Initialize the reference spectrum library: select the first spectrum that has been manually quality-checked as the initial reference spectrum and put it into the reference spectrum library;

[0017] A32. Calculate the spectral angle: starting from the second spectrum, calculate the spectral angle between each spectrum and all spectra in the reference spectrum library, and find the minimum value of the spectral angle;

[0018] A33. Determine the spectrum type: if the minimum spectrum angle is greater than a preset threshold, the spectrum is considered to be a new type of spectrum and is placed in the reference spectrum library; if the minimum spectrum angle is less than or equal to the threshold, the spectrum is considered to be similar to a spectrum type in the reference spectrum library and is not placed in the reference spectrum library;

[0019] A34. Update reference spectral library: add new types of spectra to the reference spectral library;

[0020] A35, repeating steps: repeating steps A32 to A34 for all test spectra until all spectra have been tested;

[0021] A36. Complete the construction of the reference spectral library.

[0022] Furthermore, step B specifically includes the following steps:

[0023] B1. Invalid spectrum elimination: Invalid spectra with spectral reflectance greater than 1 or less than 0 are eliminated;

[0024] B2. Spectral interpolation processing: Spectral interpolation processing is performed on the bands of 1350-1450nm, 1800-2100nm and 2400-2500nm where atmospheric absorption is serious;

[0025] B3. Spectral smoothing: Apply smoothing techniques such as spectral moving average or Savitzky-Golay filtering to eliminate random fluctuations and remove spectral noise.

[0026] Furthermore, in step B3, the specific calculation formula of the spectrum moving average method is as follows:

[0027]

[0028] Where M[i] is the moving average value of the i-th data point; K is the radius of the window, indicating that the average is performed within the range of k data points before and after the i-th data point; S[j] is the reflectance value of the j-th band.

[0029] Furthermore, in step B3, the specific steps of the Savitzky-Golay filtering method are as follows:

[0030] B31. Select the window size and polynomial order. The window size is usually an odd number, which means that m data points are selected around each data point for fitting. The polynomial order is usually an order p smaller than the window size, which is used to fit the data polynomial.

[0031] B32. Construct a local data window: For each data point yk, select a window containing m points;

[0032] B33. Fitting polynomial: In the window, use the least squares method to fit a p-order polynomial, which is expressed as:

[0033] y=a0+a1x+a2x 2 +...+a p x p (2)

[0034] Among them, the fitting coefficient A={a0,a1,a2...a p} can be calculated using the least squares method:

[0035] A=(X T X) -1 X T Y (3)

[0036] B34. Calculate the smoothed value: take the value of the fitted polynomial at the window center x=0 as the smoothed data point.

[0037] Furthermore, step C specifically includes the following steps:

[0038] C1. Spectral angle: Calculate the cosine similarity between the measured spectrum and each spectrum in the reference spectrum library as the spectral angle between the two spectra. The calculation formula for the spectral angle is as follows:

[0039]

[0040] Wherein, Α and Β are the reference spectrum and the spectrum to be evaluated respectively;

[0041] The value of the spectral angle θ ranges from 0 to π radians. The smaller the angle, the more similar the two spectra are; the larger the angle, the greater the difference between the two spectra; when θ = 0, the two spectra are exactly the same; when θ = π, the two spectra are completely opposite.

[0042] C2, Euclidean distance: Calculate the Euclidean distance between the measured spectrum and each spectrum in the reference spectrum library. The Euclidean distance calculation formula is as follows:

[0043]

[0044] n represents the number of bands, d is the Euclidean distance between the reference spectrum and the spectrum to be evaluated;

[0045] The smaller the value of the Euclidean distance d, the more similar the two vectors are; the larger the value, the greater the difference between the two vectors. When d = 0, it means that the two spectra are exactly the same;

[0046] Similarity metric: Choose an appropriate similarity metric that takes into account both the overall shape and local features of the spectrum.

[0047] Further, step D specifically includes the following steps:

[0048] D1. Threshold setting: Set the similarity threshold according to actual application requirements and historical data analysis;

[0049] D2. Anomaly marking: When the similarity between the measured spectrum and the spectrum in the reference spectrum library is lower than the set threshold, the point is marked as anomaly.

[0050] Furthermore, in step E, the report content includes information such as the location of the abnormal point, similarity value, possible cause analysis, etc. Visualization is to intuitively display the abnormal point in the form of charts and images to facilitate user understanding and analysis.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention compares the measured spectrum with a pre-established reference spectrum library, automatically identifies and marks outliers in the data, thereby improving the efficiency and accuracy of data quality auditing; the method establishes a reference spectrum library, pre-processes the measured spectrum data, calculates the similarity between the measured spectrum and the reference spectrum, detects outliers and evaluates the spectrum quality, and generates a detailed quality audit report; the method is suitable for spectral data processing in the fields of environmental monitoring, agricultural remote sensing, geological exploration, etc., and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 The present invention is a flow chart of the steps of a method for quality review of ground measurement spectral data based on a reference spectral library. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with embodiments:

[0056] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0057] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0058] The present invention provides a method for auditing the quality of ground-based spectral data based on a reference spectral library, which automatically identifies and marks abnormal points in the data by comparing the measured spectrum with a pre-established reference spectral library, thereby improving the efficiency and accuracy of data quality auditing. The method comprises the following steps:

[0059] Step 1, establish a reference spectral library: collect and organize the measured spectral data of various standard samples, and establish a reference spectral library containing a variety of typical spectral features. Sort out the spectra of clear and cloudless weather measured at field sites, remove invalid spectra, and merge similar spectra. Finally, a reference spectral library containing various typical landforms is obtained.

[0060] Step 2, preprocessing of measured spectral data: perform preprocessing operations such as denoising, smoothing, and baseline correction on the spectral data obtained from ground measurements to eliminate instrument noise and other interference.

[0061] Step 3, similarity calculation: using similarity measurement methods such as cosine similarity or Euclidean distance, the preprocessed measured spectrum is compared with the spectrum in the reference spectrum library to calculate the similarity between the two.

[0062] Step 4, outlier detection: set a similarity threshold. When the similarity between the measured spectrum and the reference spectrum is lower than the threshold, it is determined as an outlier and marked.

[0063] Step 5, generate quality audit report: Based on the anomaly marking, generate a detailed quality audit report indicating data quality issues and their locations.

[0064] Specifically, the steps are as follows:

[0065] Step 1: Create a reference spectral library:

[0066] Data collection: The spectra of standard samples measured and sorted at ground field stations are used as input spectral data.

[0067] Data collation: Standardize the collected spectral data to ensure uniform data format and wavelength range, and eliminate spectra with incorrect format or incomplete recorded content.

[0068] Construct a reference spectral library: add the sorted spectral data to the reference spectral library according to the dynamic clustering method to construct the optimal spectral library. The calculation process of constructing the reference spectral library is as follows:

[0069] 1. Initialize the reference spectrum library: Select the first spectrum that has been manually quality-checked as the initial reference spectrum and put it into the reference spectrum library.

[0070] 2. Calculate the spectral angle: Starting from the second spectrum, calculate the spectral angle between each spectrum and all spectra in the reference spectrum library, and find the minimum value of the spectral angle.

[0071] 3. Determine the spectrum type: If the minimum spectral angle is greater than a preset threshold, the spectrum is considered to be a new type of spectrum and is placed in the reference spectrum library; if the minimum spectral angle is less than or equal to the threshold, the spectrum is considered to be similar to a spectrum type in the reference spectrum library and is not placed in the reference spectrum library.

[0072] 4. Update the reference spectrum library: add new types of spectra to the reference spectrum library.

[0073] 5. Repeat steps: Repeat steps 2 to 4 for all test spectra until all spectra have been tested.

[0074] 6. Complete the construction of the reference spectral library.

[0075] Step 2, preprocessing of measured spectral data:

[0076] Invalid spectrum removal: Remove invalid spectra with spectral reflectance greater than 1 or less than 0.

[0077] Spectral interpolation processing: Spectral interpolation processing is performed on the bands of 1350-1450nm, 1800-2100nm and 2400-2500nm where atmospheric absorption is severe.

[0078] Spectral smoothing: Apply smoothing techniques such as moving average or Savitzky-Golay filtering to eliminate random fluctuations and remove spectral noise.

[0079] Among them, the spectral moving average method is a commonly used data smoothing technology, which is particularly suitable for the processing of spectral data. This method smoothes the data by calculating the average value of the spectral data within a certain window, thereby reducing the impact of noise. The specific calculation formula of the spectral moving average method is as follows:

[0080]

[0081] Where M[i] is the moving average value of the i-th data point; K is the radius of the window, indicating that the average is performed within the range of k data points before and after the i-th data point; S[j] is the reflectance value of the j-th band.

[0082] Savitzky-Golay filtering is a digital signal processing technique used for data smoothing and derivation. It achieves smoothing by fitting a polynomial to the data in a local range, and the derivative can be calculated at the same time. This filtering method is particularly useful in spectral data analysis because it can reduce noise while maintaining the characteristics of the data. The specific steps of the Savitzky-Golay filtering method are as follows.

[0083] 1. Select the window size and polynomial order. Window Size: Usually an odd number

[0084] Indicates that m data points are selected around each data point for fitting. Polynomial Order: Usually an order p smaller than the window size is selected to fit the polynomial of the data. In general, the polynomial order is 2 and the window size is 5, 7, 9 or 11.

[0085] 2. Construct a local data window: For each data point yk, select a window containing m points.

[0086] 3. Fitting polynomial. In the window, use the least squares method to fit a p-order polynomial. The p-order polynomial is expressed as

[0087] y=a0+a1x+a2x 2 +...+a p x p (2)

[0088] The fitting coefficient A={a0,a1,a2...a p} can be calculated using the least squares method

[0089] A=(X T X) -1 X T Y (3)

[0090] 4. Calculate the smoothed value. The value of the fitted polynomial at the center of the window x = 0 is used as the smoothed data point.

[0091] Savitzky-Golay filtering is a powerful data smoothing and derivation tool, especially suitable for reducing noise while retaining data features. By properly selecting the window size and polynomial order, data quality can be effectively improved and the analysis effect can be enhanced.

[0092] Step 3, similarity calculation:

[0093] Spectral Angle: Calculate the cosine similarity between the measured spectrum and each spectrum in the reference spectrum library as the spectral angle between the two spectra. Spectral Angle is a method used to measure the similarity between two spectra, and is usually used in applications such as quality review, classification, and matching of spectral data. The calculation formula for spectral angle is as follows:

[0094]

[0095] Among them, A and B are the reference spectrum and the spectrum to be evaluated respectively.

[0096] The value of the spectral angle θ ranges from 0 to π radians. The smaller the angle, the more similar the two spectra are; the larger the angle, the greater the difference between the two spectra. When θ = 0, the two spectra are exactly the same; when θ = π, the two spectra are completely opposite.

[0097] Euclidean distance: Calculate the Euclidean distance between the measured spectrum and each spectrum in the reference spectrum library. Euclidean distance is a common method to measure the difference between two vectors, which is suitable for comparison and analysis of spectral data. The formula for calculating Euclidean distance is as follows:

[0098]

[0099] n represents the number of bands, and d is the Euclidean distance between the reference spectrum and the spectrum to be evaluated.

[0100] The smaller the value of the Euclidean distance d, the more similar the two vectors are; the larger the value, the greater the difference between the two vectors. When d = 0, it means that the two spectra are exactly the same.

[0101] Similarity metric: Choose an appropriate similarity metric that takes into account both the overall shape and local features of the spectrum.

[0102] Step 4, outlier detection:

[0103] Threshold setting: Set the similarity threshold based on actual application requirements and historical data analysis.

[0104] Anomaly annotation: When the similarity between the measured spectrum and the spectrum in the reference spectrum library is lower than the set threshold, the point is marked as anomaly.

[0105] Step 5, spectral data quality review and report generation:

[0106] Report content: including the location of the anomaly, similarity value, possible cause analysis and other information.

[0107] Visualization: Anomalies are intuitively displayed in the form of charts, images, etc., making it easier for users to understand and analyze.

[0108] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for quality review of ground measurement spectral data based on a reference spectral library, characterized in that: By comparing the measured spectrum with a pre-established reference spectrum library, anomalies in the data are automatically identified and annotated, including the following steps: A. Establish a reference spectrum library: collect and organize the measured spectrum data of various standard samples, and establish a reference spectrum library containing a variety of typical spectral features; organize the spectra of clear and cloudless weather measured at field sites, remove invalid spectra, and merge similar spectra; obtain a reference spectrum library containing various typical landforms; B. Preprocessing of measured spectral data: De-noising, smoothing and baseline correction preprocessing operations are performed on the spectral data obtained from ground measurements to eliminate instrument noise and other interferences; C. Similarity calculation: Using similarity measurement methods such as cosine similarity or Euclidean distance, the preprocessed measured spectrum is compared with the spectrum in the reference spectrum library to calculate the similarity between the two; D. Outlier detection: Set a similarity threshold. When the similarity between the measured spectrum and the reference spectrum is lower than the threshold, it is determined as an outlier and marked. E. Generate quality audit report: Based on the anomaly marking, generate a detailed quality audit report indicating data quality issues and their locations.

2. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 1, characterized in that: Step A specifically comprises the following steps: A1. Data collection: Use the spectra of standard samples measured and sorted at ground field stations as the input spectral data; A2. Data collation: Standardize the collected spectral data to ensure uniform data format and wavelength range, and eliminate spectra with incorrect format or incomplete content; A3. Build a reference spectral library: Add the sorted spectral data to the reference spectral library according to the dynamic clustering method to build the optimal spectral library.

3. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 2, characterized in that: Step A3, the calculation process of building the reference spectral library is as follows: A31. Initialize the reference spectrum library: select the first spectrum that has been manually quality-checked as the initial reference spectrum and put it into the reference spectrum library; A32. Calculate the spectral angle: starting from the second spectrum, calculate the spectral angle between each spectrum and all spectra in the reference spectrum library, and find the minimum value of the spectral angle; A33. Determine the spectrum type: if the minimum spectral angle is greater than a preset threshold, the spectrum is considered to be a new type of spectrum and is placed in the reference spectrum library; If the minimum spectral angle is less than or equal to the threshold, the spectrum is considered to be similar to a spectrum type in the reference spectrum library and is not placed in the reference spectrum library; A34. Update reference spectral library: add new types of spectra to the reference spectral library; A35, repeating steps: repeating steps A32 to A34 for all test spectra until all spectra have been tested; A36. Complete the construction of the reference spectral library.

4. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 1, characterized in that: Step B specifically comprises the following steps: B1. Invalid spectrum elimination: Invalid spectra with spectral reflectance greater than 1 or less than 0 are eliminated; B2. Spectral interpolation processing: Spectral interpolation processing is performed on the bands of 1350-1450nm, 1800-2100nm and 2400-2500nm where atmospheric absorption is serious; B3. Spectral smoothing: Apply smoothing techniques such as spectral moving average or Savitzky-Golay filtering to eliminate random fluctuations and remove spectral noise.

5. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 4, characterized in that: Step B3, the specific calculation formula of the spectral moving average method is as follows: Where M[i] is the moving average value of the ith data point; K is the radius of the window, indicating that the average is performed within the range of k data points before and after the ith data point; S[j] is the reflectivity value of the j-th band.

6. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 4, characterized in that: Step B3, the specific steps of the Savitzky-Golay filtering method are as follows: B31. Select the window size and polynomial order. The window size is usually an odd number, which means that m data points are selected around each data point for fitting. The polynomial order is usually an order p smaller than the window size, which is used to fit the data polynomial. B32. Construct a local data window: For each data point yk, select a window containing m points; B33. Fitting polynomial: In the window, use the least squares method to fit a p-order polynomial, which is expressed as: y=a0+a1x+a2x 2 +...+a p x p (2) Among them, the fitting coefficient A={a0,a1,a2...a p } can be calculated using the least squares method: A=(X T X) -1 X T Y (3) B34. Calculate the smoothed value: take the value of the fitted polynomial at the window center x=0 as the smoothed data point.

7. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 1, characterized in that: Step C specifically comprises the following steps: C1. Spectral angle: Calculate the cosine similarity between the measured spectrum and each spectrum in the reference spectrum library as the spectral angle between the two spectra. The calculation formula for the spectral angle is as follows: Wherein, Α and Β are the reference spectrum and the spectrum to be evaluated respectively; The value of the spectral angle θ ranges from 0 to π radians. The smaller the angle, the more similar the two spectra are; the larger the angle, the greater the difference between the two spectra; when θ = 0, the two spectra are exactly the same; when θ = π, the two spectra are completely opposite. C2, Euclidean distance: Calculate the Euclidean distance between the measured spectrum and each spectrum in the reference spectrum library. The Euclidean distance calculation formula is as follows: n represents the number of bands, d is the Euclidean distance between the reference spectrum and the spectrum to be evaluated; The smaller the value of the Euclidean distance d, the more similar the two vectors are; the larger the value, the greater the difference between the two vectors; when d = 0, it means that the two spectra are exactly the same; Similarity metric: Choose an appropriate similarity metric that takes into account both the overall shape and local features of the spectrum.

8. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 1, characterized in that: Step D specifically comprises the following steps: D1. Threshold setting: Set the similarity threshold according to actual application requirements and historical data analysis; D2. Anomaly marking: When the similarity between the measured spectrum and the spectrum in the reference spectrum library is lower than the set threshold, the point is marked as anomaly.

9. The method for quality review of ground measurement spectral data based on a reference spectral library according to claim 1, characterized in that: In step E, the report content includes information such as the location of the anomaly point, similarity value, possible cause analysis, etc. Visualization is to intuitively display the anomaly point in the form of charts and images to facilitate user understanding and analysis.

Images