Laser radar data quality control method

Abstract

The invention provides a laser radar data quality control method. The method comprises the following steps: firstly, acquiring laser radar data; performing preliminary screening on the laser radar data through a signal-to-noise ratio threshold value, data validity and a reasonable threshold value of a wind speed value; performing time consistency check on the wind speed data by adopting a Han-Eupe filter, and eliminating or correcting singular values in a time sequence by calculating a median absolute deviation (MAD) and a standard deviation (SD); spatial autocorrelation test is carried out on wind speed data by using a Moran index, and singular points in spatial distribution are identified and eliminated by calculating a local Moran index and carrying out significance test. According to the method, the abnormal value in the wind speed data is effectively identified and processed by combining time consistency and space autocorrelation test, and the quality of the wind speed data and the reliability of an analysis result are improved. The method is suitable for the fields of meteorology, climate research, wind energy resource evaluation, environment monitoring and the like.

Description

Technical Field

[0001] The present invention relates to the field of laser radar technology, and in particular to a method for laser radar data quality control. Background Art

[0002] Coherent Doppler wind lidar has many advantages such as high measurement resolution and high measurement accuracy, and has become one of the effective tools for wind field measurement. However, due to the influence of factors such as weather conditions in the observation scene, equipment system noise, and failure to meet the uniform wind field assumption, the data quality of the radar will be affected if it is not controlled and eliminated.

[0003] There are two main types of existing lidar data quality control methods. The first is a quality control method based on radar optics and hardware systems, which ensures the accuracy of echo signals by building more stable and precise devices. The second is algorithm optimization based on inversion methods or mechanisms, which adopts targeted optimization strategies for different inversion methods. Most of these methods start from the radar itself and require a deep understanding of radar systems and algorithms. They are not suitable for users who only need to use radar data for wind field measurement-related applications. In addition, these methods ignore the temporal consistency and spatial autocorrelation of wind speed, and the degree of improvement in data quality is often limited. Summary of the invention

[0004] In order to solve the problems of the prior art, the present invention provides a method for quality control of lidar data. By combining temporal consistency and spatial autocorrelation tests, the method can effectively identify and process outliers in wind speed data, thereby improving the quality of wind speed data and the reliability of analysis results. The method is suitable for fields such as meteorology, climate research, wind energy resource assessment and environmental monitoring.

[0005] The present invention provides a method for laser radar data quality control, comprising the following steps:

[0006] 1) Obtain laser radar related data;

[0007] 2) Preliminary screening of radar data:

[0008] 2.1) Screen the upper and lower thresholds of the signal-to-noise ratio of the lidar data to reduce false wind speed alarms caused by noise in the system;

[0009] 2.2) Screening based on radar efficiency or effective status;

[0010] 2.3) Set the lower limit threshold of radial wind speed to filter small-amplitude wind speed;

[0011] 3) Use the Hampel filter to check the time consistency. For time series data with a fixed window size, calculate the median absolute deviation (MAD) value of the data segment. After converting the MAD, the standard deviation (SD) is obtained. Set a threshold n. Values ​​exceeding n times the SD are considered singular values ​​and are removed or corrected.

[0012] 4) The Moran index is used to check spatial autocorrelation, and the singular points or abnormal areas in the spatial distribution are identified through significance tests, and the singular points are eliminated or corrected.

[0013] Further improvement, the laser radar related data in step 1) includes the laser radar radial wind speed, horizontal wind speed, pitch angle, azimuth angle, range gate, signal-to-noise ratio, valid identification bit, and timestamp.

[0014] As a further improvement, the specific process of screening according to the radar efficiency or effective status described in step 2.2) is as follows: for real-time data, determine whether the data is valid based on the identification bit of the radar data, data with an identification bit of 0 is regarded as invalid data, and the invalid data is deleted; for resampled minute-level data, first calculate the proportion of valid data based on the identification bit in the real-time data, and eliminate data values ​​with an efficiency lower than 80%.

[0015] Further improvement, the specific steps of using the Hampel filter to perform time consistency check in step 3) are as follows:

[0016] 3.1) Given a radar data time series x1, x2, x3, … x at the same measurement point n And the sliding window size k, define the local median as:

[0017] m i =median(x i-k ,x i-k+1 ,…x i ,…,x i+k-1 ,x i+k)

[0018] 3.2) Calculate the median absolute deviation MAD as:

[0019] MAD i =median(|x i-k -m i |,…,|x i+k -m i |)

[0020] 3.3) Under the normal distribution assumption, the fixed relationship between median absolute deviation MAD and standard deviation SD is as follows:

[0021]

[0022] Where IQR is the interquartile range of the data;

[0023] 3.4) Identify singular points and set the threshold constant to c. When |x i -m i |>c×SD i Then determine x i is a singular value;

[0024] 3.5) Eliminate or correct singular values.

[0025] As a further improvement, the threshold constant c in step 3.4) is set to 3 by default, and the sliding window size and the threshold constant are adjusted according to the fluctuation of the radar data.

[0026] Further improvement, the specific process of removing or correcting singular values ​​in step 3.5) is to directly remove the singular values ​​from the time series, or use m i Replace singular values.

[0027] Further improvement, the specific process of using the Moran index to check spatial autocorrelation in step 4) is as follows:

[0028] 4.1) Collect lidar wind speed data after time consistency correction, and choose to re-interpolate into raster data or directly quantify the spatial relationship between measurement points according to the spatial location information of the measurement points;

[0029] 4.2) Select the local calculation space and calculate the mean wind speed in the local area

[0030]

[0031] Among them, W i is the wind speed value at the ith observation point, and n is the total number of observation points;

[0032] 4.3) Define the spatial weight matrix and use the spatial weight matrix ω ij Define the spatial relationship between the i-th observation point and the j-th observation point, including the following two ways of defining distance weights:

[0033] Inverse Distance Weighted: where d ij is the distance between i and j.

[0034] Gaussian weights: where d ij is the distance between i and j, σ is a parameter;

[0035] 4.4) Calculate the Moran index, the calculation formula is:

[0036]

[0037] Among them, W i and W j is the wind speed value at the i-th and j-th observation points; is the mean wind speed in the local area; ω ij is the spatial weight; is the sum of all spatial weights;

[0038] 4.5) Significance test of Moran's index and identification of singular points:

[0039] Significance test of Moran's index:

[0040] Calculate the Z score using the formula

[0041]

[0042] Among them, E(I i ) is the expected value of the local Moran index, var(I i ) is the variance of the local Moran's index; the Z score represents the standard deviation distance between the Moran's index and the expected value; the p-value of each region is determined by the Z-score and the standard normal distribution table. If the p-value is less than a certain significance level, the spatial autocorrelation of the region is considered to be significant;

[0043] Judgment of singular points:

[0044] Judging by threshold: According to the results of significance test, points with p-value less than a certain significance level are potential singular points;

[0045] Abnormal hot spots and cold spots: If the attribute value of an area is higher than the average value, but the attribute value of its neighboring area is lower than the average value, then the point is considered an abnormal hot spot; if the attribute value of an area is lower than the average value, but the attribute value of its neighboring area is higher than the average value, then the point is considered an abnormal cold spot. Both abnormal hot spots and cold spots are singular points.

[0046] 4.6) Eliminate or correct singular points.

[0047] Further improvement, in step 4.5), if the p-value is less than 0.05, the spatial autocorrelation of the region is considered to be significant, and the points with p-values ​​less than 0.05 are regarded as potential singular points.

[0048] Further improvement, the process of removing or correcting singular points in step 4.6) is specifically as follows: for the observation values ​​identified as singular points, these points are directly removed or replaced by the average value of the neighboring points or the interpolation method. The method of replacing the singular points is to replace the value of the singular point by the weighted average value of the surrounding points.

[0049] The beneficial effects of the present invention are:

[0050] 1. According to the error sources of radar data, the present invention controls the overall data quality of radar data from three aspects: signal-to-noise ratio, efficiency and small-amplitude wind speed, reduces the influence of errors and noise, and improves the accuracy of wind speed data.

[0051] 2. The present invention adopts the Hampel filter to check the temporal consistency of radar data and uses the Moran index to check the spatial autocorrelation, combines the spatiotemporal characteristics of wind speed, reduces the influence of extreme values ​​and abnormal points on data analysis, and makes the results more stable and reliable.

[0052] 3. In each technical process of the present invention, configurable parameters are set (for example: signal-to-noise ratio threshold, sliding window size and threshold constant of Hampel filter, weight definition in Moran index, etc.), which is convenient for configuring the best parameters according to specific problems and has wide adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is the overall flow chart of the present invention.

[0055] Figure 2 This is the flow chart of time consistency check.

[0056] Figure 3 This is an example graph of the Moran coefficient.

[0057] Figure 4 This is the flow chart of spatial autocorrelation test. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] A specific process of the present invention is as follows Figure 1 As shown, the following steps are included:

[0060] Step 1: Obtain lidar related data.

[0061] Collect lidar radial wind speed, horizontal wind speed, pitch angle, azimuth angle, range gate, signal-to-noise ratio, valid identification bit, timestamp and other information.

[0062] Step 2: Preliminary screening of radar data.

[0063] LiDAR data requires the following three preliminary screenings:

[0064] 1. Reduce false alarms of wind speed caused by noise in the system. The signal detected by the coherent Doppler lidar contains useful echo signals and noise signals. The signal-to-noise ratio (SNR) is an important indicator to measure the detection capability and anti-noise capability of the system. Low signal-to-noise ratio may cause false alarms of non-real wind speed signals. Therefore, before obtaining lidar data, it is necessary to ensure that the data has been screened by the upper and lower thresholds of the signal-to-noise ratio.

[0065] 2. Filter according to radar efficiency or effective status. For real-time data, the validity of the data should be determined based on the identification bit of the radar data. Data with an identification bit of 0 is considered invalid data and deleted. For resampled minute-level data, the proportion of valid data (i.e., efficiency) should be calculated based on the identification bit in the real-time data, and data values ​​with an efficiency of less than 80% should be eliminated. When the efficiency is less than 80%, the sampling mean will be biased, introducing a large error.

[0066] 3. Filter small-amplitude wind speeds. In radar data, some very small wind speeds may appear due to various reasons. For example, for the PPI scanning mode, the radial wind speed of the lidar can be expressed as

[0067] v R (β)=-v h (β)cos(β-β0)cosα

[0068] Among them, v R is the radar radial wind speed, v h is the horizontal wind speed in the wind field, β is the azimuth, β0 is the angle to the north, and α is the elevation angle. When β-β0=90°, cos(β-β0) is approximately 0, and the radial wind speed is a small value. When it is substituted into the VAD operation, it may not converge, resulting in a large deviation in the inverted wind speed and direction. Therefore, the lower limit threshold of the radial wind speed is set, and the data is discarded when it is less than 0.5m / s to ensure the overall inversion accuracy.

[0069] Step 3: Use Hampel filter to check the time consistency, such as Figure 2 shown.

[0070] The Hampel filter is an algorithm for detecting singular values ​​in time series. For time series data with a fixed window size, the median absolute deviation (MAD) value of the data segment is calculated, and the standard deviation (SD) is obtained by converting the MAD. The threshold n is set, and values ​​exceeding n times the SD are considered singular values. The specific steps are as follows:

[0071] Given a radar data time series x1,x2,x3,…x at the same measurement point n And the sliding window size k, define the local median as:

[0072] m i =median(x i-k ,x i-k+1 ,…x i ,…,x i+k-1 ,x i+k)

[0073] (2) In statistics, the Median Absolute Deviation (MAD) is a measure of statistical dispersion. It is a robust measure of the variability of univariate samples of quantitative data. The MAD is calculated as:

[0074] MAD i =median(|x i-k -m i |,…,|x i+k -m i |)

[0075] (3) Under the normal distribution assumption, there is a fixed relationship between MAD and SD:

[0076]

[0077] In a normal distribution, the interquartile range (IQR, the distance between the third quartile and the first quartile) of the data is approximately 1.349 times the standard deviation. Since the MAD is based on the median, it is approximately half of the IQR, so

[0078]

[0079] (4) Identify singular points. Set the threshold constant to c, when |x i -m i |>c×SD i Then determine x i The threshold constant c is generally set to 3, but the sliding window size and threshold constant can be adjusted appropriately according to the fluctuation of radar data.

[0080] (5) Eliminate or correct singular values. For singular values, you can choose to directly remove them from the time series, or use m i Replace singular values.

[0081] In the process of preprocessing lidar data using the Hampel filter, this method uses the median instead of the mean, which can largely avoid the influence of extreme values ​​on the data preprocessing effect and retain the original fluctuation characteristics of the lidar data. Compared with other outlier processing methods, the data processing effect and robustness are better.

[0082] Step 4: Use Moran's I to check spatial autocorrelation.

[0083] Moran's I is a correlation coefficient that measures the overall spatial autocorrelation of a data set. In other words, it measures how similar an object is to other objects around it. Spatial autocorrelation is multi-directional and multi-dimensional, so it can be used to find patterns in complex data sets. It is similar to the correlation coefficient, with values ​​ranging from -1 to 1, such as Figure 3 However, while the other coefficients measure perfect correlation versus no correlation, Moran's coefficient is slightly different (due to more complicated spatial calculations):

[0084] Figure 3 Part (a) of : -1 is a perfect clustering of different values ​​(which can be considered perfect discretization).

[0085] Figure 3 Part (b) of : 0 means no autocorrelation (perfect randomness).

[0086] Figure 3 Part (c) in : +1 indicates perfect clustering of similar values.

[0087] The following are the steps to use the Moran index to check the spatial autocorrelation of wind speed data. Figure 4 As shown:

[0088] (1) Collect the time-corrected wind speed data from the lidar. Based on the spatial location information of the measurement points, you can choose to re-interpolate it into raster data, or you can directly quantify the spatial relationship between the measurement points. Raster data is a form of representation of geographic spatial data that divides the surface into pixels or cells of uniform size and uses numerical values ​​to describe the attributes of each pixel.

[0089] (2) Select a local calculation space and calculate the mean wind speed within the local range

[0090]

[0091] Among them, W iis the wind speed value at the ith observation point, and n is the total number of observation points.

[0092] (3) Define the spatial weight matrix. Spatial weight matrix ω ij The spatial relationship between i and j is defined. The following two distance weight definitions can be selected:

[0093] Inverse Distance Weighted: where d ij is the distance between i and j.

[0094] Gaussian weights: where d ij is the distance between i and j, and σ is a parameter.

[0095] (4) Calculate the Moran index, the calculation formula is:

[0096]

[0097] Among them, W i and W j is the wind speed value at the i-th and j-th observation points; is the mean wind speed in the local area; ω ij is the spatial weight; is the sum of all spatial weights.

[0098] (5) Significance test of Moran's index and identification of singular points. Calculate the Z score, the formula is:

[0099]

[0100] Among them, E(I i ) is the expected value of the local Moran index, var(I i ) is the variance of the local Moran's index. The Z score represents the standard deviation distance between the Moran's index and the expected value. The p-value of each region can be determined by the Z score and the standard normal distribution table. If the p-value is less than a certain significance level (such as 0.05), the spatial autocorrelation of the region is considered significant. Judgment of singular points:

[0101] Judge by threshold: According to the results of the significance test, a threshold is set to identify singular points. Points with p-values ​​less than 0.05 can be selected as potential singular points.

[0102] Abnormal hot spots and cold spots: If the attribute value of an area is higher than the average, but the attribute value of its neighboring areas is lower than the average, then the point is considered an abnormal hot spot; if the attribute value of an area is lower than the average, but the attribute value of its neighboring areas is higher than the average, then the point is considered an abnormal cold spot.

[0103] Points that meet one of the above two conditions are considered singular points.

[0104] (6) Eliminate or correct singular points. For observations identified as singular points, you can choose to directly eliminate these points. If you do not want to eliminate the data, you can use the average value of the neighboring points or interpolation methods to replace these singular points. For example, you can use the weighted average value of the surrounding points to replace the value of the singular point.

[0105] (7) Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, the above is only a preferred implementation of the present invention. Since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The above is only a specific implementation of the present invention, but the protection scope of the present invention is not limited to this. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention without departing from the principle of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for laser radar data quality control, characterized in that The steps include: 1) Obtain laser radar related data; 2) Preliminary screening of radar data: 2.1) Screen the upper and lower thresholds of the signal-to-noise ratio of the lidar data to reduce false wind speed alarms caused by noise in the system; 2.2) Screening based on radar efficiency or effective status; 2.3) Set the lower limit threshold of radial wind speed to filter small-amplitude wind speed; 3) Use Hampel filter to check time consistency. For time series data with fixed window size, calculate the median absolute deviation MAD value of the data segment, convert the absolute deviation MAD to get the standard deviation SD, set the threshold n, and the value exceeding n times the standard deviation SD is regarded as a singular value, and the singular value is removed or corrected; 4) The Moran index is used to check spatial autocorrelation, and the singular points or abnormal areas in the spatial distribution are identified through significance tests, and the singular points are eliminated or corrected.

2. The method for laser radar data quality control according to claim 1, characterized in that: The laser radar related data in step 1) includes the laser radar radial wind speed, horizontal wind speed, pitch angle, azimuth angle, range gate, signal-to-noise ratio, effective identification bit, and timestamp.

3. The method for laser radar data quality control according to claim 1, characterized in that: The specific process of screening according to the radar efficiency or effective status described in step 2.2) is as follows: for real-time data, determine whether the data is valid based on the identification bit of the radar data, and data with an identification bit of 0 is regarded as invalid data, and the invalid data is deleted; for resampled minute-level data, first calculate the proportion of valid data based on the identification bit in the real-time data, and eliminate data values ​​with an efficiency lower than 80%.

4. The method for laser radar data quality control according to claim 1, characterized in that: Step 3) The specific steps of using the Hampel filter to perform time consistency check are as follows: 3.1) Given a radar data time series x1, x2, x3, … x at the same measurement point n And the sliding window size k, define the local median as: m i =median(x i-k ,x i-k+1 ,…x i ,…,x i+k-1 ,x i+k) 3.2) Calculate the median absolute deviation MAD as: MAD i =median(|x i-k -m i |,…,|x i+k -m i |) 3.3) Under the normal distribution assumption, the fixed relationship between median absolute deviation MAD and standard deviation SD is as follows: Where IQR is the interquartile range of the data; 3.4) Identify singular points and set the threshold constant to c. When |x i -m i |>c×SD i Then determine x i is a singular value; 3.5) Eliminate or correct singular values.

5. The method for laser radar data quality control according to claim 4, characterized in that: In step 3.4), the threshold constant c is set to 3 by default, and the sliding window size and threshold constant are adjusted according to the fluctuation of radar data.

6. The method for laser radar data quality control according to claim 4, characterized in that: Step 3.5) The specific process of removing or correcting singular values ​​is to directly remove the singular values ​​from the time series, or use m i Replace singular values.

7. The method for laser radar data quality control according to claim 1, characterized in that: The specific process of using the Moran index to check spatial autocorrelation in step 4) is as follows: 4.1) Collect lidar wind speed data after time consistency correction, and choose to re-interpolate into raster data or directly quantify the spatial relationship between measurement points according to the spatial location information of the measurement points; 4.2) Select the local calculation space and calculate the mean wind speed in the local area Among them, W i is the wind speed value at the ith observation point, and n is the total number of observation points; 4.3) Define the spatial weight matrix and use the spatial weight matrix ω ij Define the spatial relationship between the i-th observation point and the j-th observation point, including the following two ways of defining distance weights: Inverse Distance Weighted: where d ij is the distance between i and j; Gaussian weights: where d ij is the distance between i and j, σ is a parameter; 4.4) Calculate the Moran index, the calculation formula is: Among them, W i and W j is the wind speed value at the i-th and j-th observation points; is the mean wind speed in the local area; ω ij is the spatial weight; is the sum of all spatial weights; 4.5) Significance test of Moran's index and identification of singular points: Significance test of Moran's index: Calculate the Z score using the formula Among them, E(I i ) is the expected value of the local Moran index, var(I i ) is the variance of the local Moran's index; the Z score represents the standard deviation distance between the Moran's index and the expected value; the p-value of each region is determined by the Z-score and the standard normal distribution table. If the p-value is less than a certain significance level, the spatial autocorrelation of the region is considered to be significant; Judgment of singular points: Judging by threshold: According to the results of significance test, points with p-value less than a certain significance level are potential singular points; Abnormal hot spots and cold spots: If the attribute value of an area is higher than the average value, but the attribute value of its neighboring area is lower than the average value, then the point is considered an abnormal hot spot; if the attribute value of an area is lower than the average value, but the attribute value of its neighboring area is higher than the average value, then the point is considered an abnormal cold spot. Both abnormal hot spots and cold spots are singular points. 4.6) Eliminate or correct singular points.

8. The method for laser radar data quality control according to claim 7, characterized in that: In step 4.5), if the p-value is less than 0.05, the spatial autocorrelation of the region is considered significant, and the points with p-values ​​less than 0.05 are regarded as potential singular points.

9. The method for laser radar data quality control according to claim 7, characterized in that: The process of removing or correcting singular points in step 4.6) is specifically as follows: for observations identified as singular points, these points are directly removed or the average value of neighboring points or an interpolation method is used to replace these singular points.

10. The method for laser radar data quality control according to claim 9, characterized in that: The method for replacing the singular point is to replace the value of the singular point by the weighted average value of the surrounding points.

Images