A method for obtaining an equivalent surface reflectance curve based on a normal distribution

Abstract

The application discloses a normal distribution-based equivalent ground reflectivity curve acquisition method, and belongs to the field of space optical remote sensing. The method comprises the following steps: using a spectrometer to measure the ground to obtain measurement data; reading the measurement data to obtain a measurement curve; intercepting the average value of the reflectivity of a relatively stable wavelength interval as the description value of the measurement curve, processing all the measurement curves to obtain a list list1 of the description values; through continuous clustering, counting the classes with higher frequency, and obtaining the maximum value and the minimum value of the class descriptors; counting all the description values of list1 between the maximum value and the minimum value to obtain list3, and calculating the average value a and the standard deviation sigma of list3; the curve corresponding to the descriptor in list1 within the range of [a-1.5*sigma, a+1.5*sigma] is a reserved spectral curve. The method can more effectively eliminate various error data in the reflectivity measurement process, and improve the absolute radiation calibration accuracy.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of space optical remote sensing satellite data preprocessing, and particularly relates to a normal distribution-based equivalent ground reflectivity curve acquisition method and system and an electronic device. BACKGROUND

[0002] At present, a reflectivity-based method is used for site substitution calibration of an optical remote sensing satellite sensor. When the reflectivity-based method is used for absolute radiation calibration of each waveband of the optical remote sensing satellite sensor, equivalent ground reflectivity corresponding to the waveband is needed. The ground reflectivity is generally obtained by manually measuring the reflectivity of a ground target by using a spectrometer during satellite overflight calibration of a site. One point is measured every 50 m in the site, and 5 measurements are performed on the ground target at each measurement point. Before and after the measurements, the reference white board is measured once. During the measurement process, the following situations are prone to occur: the white board is measured by mistake, the measurement angle is not perpendicular to the white board, or the reflectivity of other non-target objects is measured. Therefore, the measured curve needs to be processed to obtain the equivalent ground reflectivity.

[0003] From the published articles and public data, in the process of calculating the equivalent reflectivity, the measured original spectral data is analyzed mainly by manual observation of the reflectivity curve shape, deletion of obvious abnormal points, and then average of the spectral data collected near each measurement point. In this process, only gross errors, that is, obvious non-ground reflectivity curves, are considered. Another method is to remove error reflectivity based on the root mean square error. This method is not used in the process of measuring the ground reflectivity of the optical remote sensing satellite sensor. Some researchers use this method in the process of measuring the ground reflectivity, but the method has not been published.

[0004] The related technologies are as follows:

[0005] 1. "Resource 3 Multi-Spectral Sensor Site Radiation Calibration and Verification" (Zhang Xuewen, Spectroscopy and Spectral Analysis, September 2014)

[0006] Before data processing, the measured original spectral data is analyzed, bad points and abnormal points with large differences are deleted, then the spectral data collected near each measurement point is averaged, and finally the spectral reflectivity of the sampling point is obtained.

[0007] 2. "High-Frequency Radiation Calibration of Wide-Field Imager of Gaofen-1 Satellite" (Han Qijin, Optics Precision Engineering, July 2014)

[0008] According to the difference of the measuring instrument, the measured spectral data set is processed and analyzed, the spectral data measured multiple times in each year is averaged, and the site reflectivity spectrum is obtained.

[0009] After analyzing the related art, it is found that the prior art has the following problems:

[0010] (1) In the calibration process of an optical remote sensing satellite sensor, the measurement value of the ground reflectivity curve is not considered to have errors or only the coarse error caused by repeated measurement of a white board is considered. The error caused by measurement of the clothes of the measurer, ground shadow and non-vertical measurement is not considered and effectively processed. These non-effective measurement values can cause the ground equivalent band reflectivity to not truly reflect the radiation performance of the optical remote sensing satellite sensor, thereby causing calibration error.

[0011] (2) The prior art lacks a robust and reliable method for automatically removing non-effective measurement values from a set of ground reflectivity curves measured in the absolute radiation calibration process of an optical remote sensing satellite sensor, thereby improving the accuracy of the equivalent band reflectivity. SUMMARY

[0012] The present application proposes an equivalent ground reflectivity curve acquisition method based on a normal distribution to improve the accuracy of the equivalent band reflectivity of hyperspectral ground reflectivity, thereby improving the accuracy of the radiation calibration of an on-orbit satellite remote sensor, in view of the development status and existing problems of the prior art.

[0013] The present application adopts the following technical solution:

[0014] First step: a spectrometer is used to measure the ground to obtain measurement data, and the measurement is repeated n1 times, n1 being a positive integer. Then, the reflectivity column in the measurement data is taken as a reflectivity curve set, and the measurement curves included in the reflectivity curve set are n1.

[0015] Second step: due to the measurement characteristics of the reflectivity spectrometer, the average value of the reflectivity in the wavelength interval 360-900 nm, 1019-1328 nm, 1439-1796 nm and 1990-2370 nm, which is relatively stable, is taken as the description value di of the measurement curve, i∈[1-n1]. The description values di of all measurement curves are processed to obtain the list descriptor_list1=[d1,d2……dn1]. n1 ]。

[0016] Third step: the list descriptor_list1 is clustered with 1 (corresponding to the reflectivity 0.01) as the distance to obtain a dictionary D1 with the frequency as the key and the corresponding class list as the value. The descriptor of the class with the highest frequency is taken to form the list descriptor_list2=[d1,d2……dn1]. n2 ]。

[0017] Fourth step: descriptor list descriptor_list2 is clustered again with a distance of 0.25, a dictionary D2 is obtained with frequency as key and corresponding class list as value, the classes with frequency greater than 5% of the total records are counted, and the maximum and minimum values of the descriptors of these classes are found.

[0018] Fifth step: in the descriptor value list descriptor_list1, all the descriptor values between the maximum and minimum values obtained in the fourth step are counted to obtain descriptor_list3=[d1, d2……d n3 ] The average value a and the standard deviation σ of descriptor_list3 are calculated.

[0019] Sixth step: the curve corresponding to the descriptors in [a-1.5×σ, a+1.5×σ] range in descriptor_list1 is taken out as the reserved spectrum curve, and the average value of the reserved spectrum curve is taken as the final standard reflectivity curve.

[0020] The standard deviation

[0021] The whiteboard-corrected standard reflectivity curve is obtained from the standard reflectivity curve.

[0022] ρ(λ)=ρ’(λ)×ρ s (λ) (1)

[0023] Wherein, ρ(λ): the whiteboard-corrected standard reflectivity curve; ρ’(λ) is the standard reflectivity curve; ρ s (λ) represents the reflectivity curve of the reference whiteboard. When the wavelength λ takes different values, the reflectivity-ρ’(λ), ρ’(λ), ρ’(λ) satisfies the above relationship.

[0024] In the formula, ρ s (λ) represents the reflectivity of the reference whiteboard, which is usually the result of laboratory calibration;

[0025] ρ’(λ)=v(λ) / v(λ), v(λ) represents the measured value of the ground object, v s (λ) represents the measured value of the reference whiteboard;

[0026] The original measurement data file of the spectrometer includes wavelength column, reference whiteboard radiation brightness value column, ground object radiation brightness measurement value column, and ratio column (reflectivity).

[0027] An equivalent ground surface reflectivity curve acquisition system based on normal distribution, comprising

[0028] A measurement module, which obtains measurement data by measuring the ground surface n1 times and sends the measurement data to the reading processing module;

[0029] The reading and processing module reads the wavelength and reflectance columns of the spectrometer measurement data to obtain n1 measurement curves.

[0030] The average reflectance of the wavelength range where the spectrometer response is relatively stable is taken as the descriptor value d of the measurement curve. The above processing is performed on all measurement curves to obtain a list of descriptor values ​​d, descriptor_list1.

[0031] The removal module performs clustering on descriptor_list1, counting those classes whose frequency is greater than 5% of the total number of records in the class, and obtaining the maximum and minimum values ​​of the descriptors of these classes; all descriptor values ​​in descriptor_list1 between the maximum and minimum values ​​are counted to obtain descriptor_list3 = [d1, d2, ..., dn3], and the mean a and standard deviation σ of descriptor_list3 are calculated; the curves corresponding to the descriptors in descriptor_list1 in the range of [a-1.5×σ, a+1.5×σ] are taken as the retained spectral curves.

[0032] An electronic device, including a memory and a processor,

[0033] Memory: Used to store computer-readable instructions; and

[0034] A processor for executing the computer-readable instructions to perform the method described in any of the preceding descriptions.

[0035] In summary, this application includes at least the following beneficial technical effects:

[0036] The method proposed in this application can more effectively eliminate various error data and erroneous reflectance curves during reflectance measurement, thereby improving the accuracy of absolute radiometric calibration. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the method in an embodiment of this application;

[0038] Figure 2 The method in the embodiments of this application processes the reflectance curve dataset results. Detailed Implementation

[0039] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments:

[0040] This application discloses a method for obtaining equivalent surface reflectance curves based on normal distribution. The specific implementation involves program code. Taking the SVC measurement data from the Dunhuang calibration field on August 12, 2020, as an example, this data contains 177 measurement data points. (Refer to...)Figure 1 This includes the following steps:

[0041] Step 1: Use a spectrometer to measure the landmark to obtain measurement data, and repeat the test n1 times. In this embodiment, n1 is 177. Then, read the original measurement data file of the spectrometer. Specifically, read the .sig files in the folder one by one to obtain 177 measurement curves. Read the wavelength column and ratio column (the ratio column is the reflectance) of the measurement curves in the relatively stable wavelength ranges of 360-900nm, 1019-1328nm, 1439-1796nm, and 1990-2370nm.

[0042] Step 2: Calculate the average reflectance ratio d of each measurement curve as the descriptor di of that measurement curve, i∈[1-177]. Perform the above processing on all 2020 measurement curves to obtain a list of descriptors di: descriptor_list1=[25.3281,…,97.1896,…,24.5655].

[0043] Step 3: Cluster the list descriptor_list1 with a distance of 1 (corresponding to reflectivity 0.01) to obtain a dictionary D1 with the frequency as the key and the corresponding class list as the value. Take the descriptors of the class with the highest frequency of 174 to form a list descriptor_list2 = [25.3281, 24.9783, ..., 24.5655].

[0044] Step 4: Cluster the descriptor list `descriptor_list2` again at a distance of 0.25, resulting in a dictionary `D2` with frequencies as keys and corresponding class lists as values, as shown in the table. Count the classes whose frequencies are greater than 5% of the total records, and find the maximum value (25.4424) and minimum value (23.6924) of these class descriptors.

[0045] D2 Statistical Table

[0046]

[0047]

[0048] Step 5: In the descriptor_list1, collect all the descriptor values ​​between the maximum and minimum values ​​obtained in step 4 to obtain descriptor_list3. Calculate the mean a = 24.4995 and the standard deviation σ = 0.7024 of descriptor_list3.

[0049] Step 6: Extract the spectral curves corresponding to descriptors in the range [a-1.5×σ, a+1.5×σ] from descriptor_list1, that is, the descriptor values ​​in [23.4459, 25.5531], as the retained spectral curves. Calculate the average value of the retained spectral curves as the final standard reflectance curve.

[0050] The final deletion curve is as follows Figure 2 As shown, red represents the deleted curve, yellow represents the retained spectral curve, and green represents the final calculated standard reflectance curve.

[0051] ρ'(λ) is obtained from the standard reflectance curve.

[0052] Then, using formula (1), the spectral reflectance ρ(λ) of the ground object at the sampling point is obtained:

[0053] ρ(λ)=ρ'(λ)×ρ s (λ) (1)

[0054] In the formula, ρ s (λ) represents the reflectivity of the reference whiteboard, which is usually the result of laboratory calibration;

[0055] ρ'(λ) = v(λ) / v(λ), where v(λ) represents the measured value of the ground target, v s (λ) represents the measurement value of the reference whiteboard;

[0056] The raw measurement data file of the spectrometer includes wavelength columns, reference whiteboard measurement columns, ground target measurement columns, and ratio columns (reflectivity).

[0057] Application Examples

[0058] This application has been applied to the "Civil and Commercial Calibration Data Transfer and Processing System" to process the raw measurement data of the Dunhuang calibration field spectrometer. Taking the 2020 measurement data of the Dunhuang calibration field spectrometer as an example, the effectiveness of the technology in this application will be explained.

[0059] Verification instructions

[0060] The required reflectance dataset was collected at the Dunhuang Satellite Radiometric Correction Field (hereinafter referred to as the Dunhuang Field). The Dunhuang Field is a major site for radiometric calibration of my country's Earth observation remote sensing satellites and an ideal test area for conducting on-orbit absolute radiometric correction experiments. The geographical location of the center of the uniform region is N40°05′27.75″, E94°23′39″. The area of ​​the reflectance data collection area for verification is 300m × 300m, with a central altitude of 1229m. Using a continuous spectrometer in reflectance measurement mode, the measured, displayed, and stored spectra were automatically normalized to the latest measured reference board signal; this value is the relative reflectance factor. Within the 300m × 300m field, measurements were taken at 50m intervals, for a total of 36 points. Each measurement point was used to measure the surface target 5 times, and the reference white board was measured once before and after each measurement. GPS was used for positioning and tracking during the measurement process to obtain accurate surface positioning data. The validation data consisted of multiple sets of reflectance datasets measured by the China Resources Satellite Center at the Dunhuang radiation calibration field in 2019 and 2020, as well as Landsat 8 data that passed overhead simultaneously on the day the reflectance was collected. Atmospheric parameter data were measured by the solar photometer and microwave radiometer at the Dunhuang calibration field.

[0061] ①Verification Method Description

[0062] The verification method employed was a comparative verification approach. To verify that the proposed method could accurately eliminate errors in manually measured reflectance data from the calibration field and improve calibration accuracy, the experiment used the direct averaging method, the root mean square error elimination method, and the proposed method to process the field-measured reflectance dataset. The retained reflectance curves were averaged, and then Equation 1 was used to obtain the equivalent band spectral reflectance at the sampling point. The equivalent band spectral reflectance calculated using different algorithms was input into the radiative transfer model, with atmospheric parameters using measured data from the Dunhuang field as input. The calibration coefficients obtained using different algorithms were compared with the gain coefficients calibrated on the Landsat 8 satellite. The absolute value of the relative error between the Landsat 8 calibration gain coefficient and the calibration coefficients calculated by various elimination algorithms was used as the evaluation index; algorithms with smaller relative errors in the calibration gain coefficient were considered superior.

[0063]

[0064] Where a' represents the scaling coefficients obtained by different methods, and a represents the Landsat8 scaling coefficient.

[0065] The direct averaging method involves identifying all data from the current measurement and manually removing measurement curves with significant abnormal deviations. The remaining curves are then used as a calculation curve set. The average value of this curve set is calculated for each wavelength to obtain a wavelength-reflectivity curve, which is then used as the standard curve for this measurement.

[0066] The root mean square error (RMSE) elimination method involves identifying all data from the current measurement. The identification method is as follows: For each curve, calculate the RMSE Rms of the ten curves following it, including the current curve, forming a list {Rms}. Then, take the average of this list, Rms. ave As a criterion, the root mean square error of all curves is greater than Rms. ave All curves were discarded, and the average of the remaining curves was calculated to obtain a wavelength-reflectivity curve, which was then used as the standard curve for this measurement.

[0067] ②Verification results

[0068] The table below lists the site calibration gain coefficient 'a' and the Landsat8 calibration gain coefficient 'a' obtained by the method of this application, the direct averaging method, and the root mean square error calculation, respectively.

[0069] Table 2 shows the scaling gain coefficient 'a' obtained by the three algorithms.

[0070]

[0071]

[0072] Table 3 shows the relative calibration errors of each band calculated by the method of this application, the direct averaging method, and the root mean square error method.

[0073] Table 3 shows the relative error results of the scaling gain coefficient 'a' obtained by three different algorithms.

[0074]

[0075]

[0076] ③ Analysis and Conclusion of Verification Results

[0077] As shown in the table, the average relative error of the calibration coefficients calculated from the equivalent reflectance obtained by the method of this application is 1.97%, which is better than the average relative error of 2.15% of the root mean square error method and much higher than the average relative error of 5.44% of the direct averaging method. The results indicate that the algorithm proposed in this application effectively eliminates erroneous reflectance curves, significantly reduces the relative error compared to the direct averaging method, and improves the accuracy of absolute radiometric calibration. This demonstrates that the method proposed in this application can more effectively eliminate various error data during reflectance measurement.

[0078] The above are preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made to the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for obtaining equivalent surface reflectance curves based on normal distribution, characterized in that: It includes using a spectrometer to make n1 measurements on the ground surface to obtain measurement data; reading the wavelength column and reflectivity column in the measurement data as the horizontal and vertical coordinates respectively to plot n1 measurement curves; respectively intercepting the average value of the reflectivity in the wavelength range where the spectrometer response is stable for each of the n1 measurement curves as the description value di of this measurement curve, and obtaining a description list list1 of the n1 measurement curves; performing at least two consecutive clusterings on list1, and statistically determining the classes with a frequency greater than 5-10% of the total number of records in the class as the selected classes, and obtaining the maximum and minimum values of the description values of the selected classes; statistically determining all the description values of list1 between the maximum and minimum values to obtain list3, and calculating the average value a and standard deviation σ of list3; extracting the curves corresponding to the descriptors in the range of [a - 1.5×σ, a + 1.5×σ] in list1 as the retained spectral curves, that is, the equivalent ground surface reflectivity curves; The clustering is two consecutive clusterings, specifically including clustering list1 with x1 as the distance, and forming a list list2 with the descriptors of the class with the highest frequency; clustering list2 again with x2 as the distance, where x2 < x1, and statistically determining the classes with a frequency greater than 5-10% of the total number of records in the class as the selected classes, and obtaining the maximum and minimum values of the descriptors of the selected classes; The x1 is 0.8 - 1.2; the x2 is 0.2 - 0.

3.

2. The method for obtaining the equivalent surface reflectance curve based on normal distribution according to claim 1, characterized in that: Taking the average value of the retained spectral curves as the standard reflectivity curve.

3. The method for obtaining the equivalent surface reflectance curve based on normal distribution according to claim 2, characterized in that: Obtaining the standard reflectivity curve corrected by the whiteboard from the standard reflectivity curve, ρ(λ) = ρ’(λ) × ρs(λ) In the formula, ρ(λ): the standard reflectivity curve corrected by the whiteboard; ρ’(λ) is the standard reflectivity curve; ρs(λ) represents the reflectivity curve of the reference whiteboard.

4. A method for obtaining equivalent surface reflectance curves based on normal distribution according to any one of claims 1-3, characterized in that: Statistically determining the classes with a frequency greater than 5% of the total number of records in the class as the selected classes.

5. The method for obtaining the equivalent surface reflectance curve based on normal distribution according to claim 4, characterized in that: The x1 is 1.

6. The method for obtaining the equivalent surface reflectance curve based on normal distribution according to claim 4, characterized in that: The x2 is 0.

25.

7. The method for obtaining the equivalent surface reflectance curve based on normal distribution according to claim 1, characterized in that: The wavelength range where the spectrometer response is stable is 360 - 900nm, 1019 - 1328nm, 1439 - 1796nm, 1990 - 2370nm.

8. A system for obtaining equivalent surface reflectance curves based on normal distribution, characterized in that: A method for obtaining an equivalent ground surface reflectivity curve based on normal distribution according to any one of claims 1 - 7, including a measurement module that makes n1 measurements on the ground surface to obtain measurement data and sends the measurement data to a reading and processing module; A reading and processing module that reads the wavelength column and reflectivity column of the measurement data to obtain n1 measurement curves, respectively intercepts the average value of the reflectivity in the wavelength range where the spectrometer response is stable for each of the n1 measurement curves as the description value di of this measurement curve, and obtains a description list list1 of the n1 measurement curves; The elimination module clusters the description list list1, selecting the classes whose frequencies are greater than 5-10% of the total number of records in each class as the selection classes, and obtaining the maximum and minimum values ​​of the descriptors in the selection classes; all description values ​​in list1 between the maximum and minimum values ​​are counted to obtain list3, and the mean a and standard deviation σ of list3 are calculated; the curves corresponding to the descriptors in list1 within the range of [a-1.5×σ, a+1.5×σ] are taken as the retained spectrum curves.

9. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is for storing computer-readable instructions; and the processor is for executing the computer-readable instructions to perform the method as described in any one of claims 1 to 7.

Images