Data quality control method of atmospheric vertical profile measurement system

Abstract

The invention discloses a data quality control method of an atmospheric vertical profile measurement system. The method comprises the following steps: step 1, carrying out same-address synchronous observation, measurement and data recording on equipment in the system; step 2, after the measured and recorded data reach a certain time length, performing time consistency control on the data; 3, after the data is measured and recorded for a certain time length, space consistency control is carried out on the data; step 4, performing quality control based on a data historical extreme value; 5, verifying the consistency of the internal data of the system; step 6, performing associated quality control on the same type of data; and step 7, performing quality identification on the data after quality control. According to the method disclosed by the invention, the feasibility and effectiveness of controlling the data quality of the atmospheric vertical profile measurement system are verified, the data product quality can be improved, and the credibility of the detection data of the atmospheric vertical profile measurement system is higher, so that support is provided for meteorological monitoring and forecasting, high-precision weather monitoring and forecasting are realized, and the method is suitable for popularization and application. The method has very important significance in the fields of spaceflight measurement and control and the like, and the method can also be expanded and applied to all atmosphere vertical profile comprehensive measurement systems.

Description

Technical Field

[0001] The present invention belongs to the field of tropospheric atmospheric remote sensing detection, and particularly relates to a data quality control method for an atmospheric vertical profile measurement system in this field. The atmospheric vertical profile measurement system is composed of a microwave radiometer, a laser wind measurement radar, a millimeter wave cloud radar, and the like. Background Art

[0002] The atmospheric vertical profile measurement system is generally composed of laser wind measuring radar, microwave radiometer, millimeter wave cloud radar and ground-based remote sensing detection data fusion and analysis software. By integrating multiple equipment such as laser wind measuring radar, microwave radiometer and millimeter wave cloud radar for networked collaborative observation, it is used for refined and continuous observation of the vertical profiles of meteorological elements such as atmospheric temperature, humidity, wind direction and speed, clouds, water vapor, etc. in the troposphere, which can enhance the refined monitoring and early warning capabilities of important weather such as haze, freezing, and thunderstorms.

[0003] Currently, individual devices such as laser wind radars, microwave radiometers, and millimeter-wave cloud radars generally have device-level data quality control capabilities, ensuring the quality of their basic data products. However, for atmospheric vertical profile measurement systems, the temporal and spatial resolution and accuracy of each device vary significantly, requiring quality control to achieve effective data fusion.

[0004] Numerical weather forecast models rely on high-quality profile data for initialization. Outliers can cause model divergence. Therefore, the measurement results of the atmospheric vertical profile measurement system need to be highly credible and scientifically rigorous. Data that has not been quality-controlled may lead to unreliable research conclusions and even cause errors in weather forecasts. Therefore, data quality control of the atmospheric vertical profile measurement system is a core link to ensure the scientific value and practicality of the data. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a data quality control method for an atmospheric vertical profile measurement system, so as to improve the quality of data products and solve the problem of low reliability of detection data of the atmospheric vertical profile measurement system.

[0006] The present invention adopts the following technical solutions:

[0007] A data quality control method for an atmospheric vertical profile measurement system, the improvement of which comprises the following steps:

[0008] Step 1: All devices in the system perform co-located synchronous observation, measurement and data recording;

[0009] Step 2: After measuring and recording data for a certain period of time, perform time consistency control on the data;

[0010] Step 3: After measuring and recording data for a certain period of time, perform spatial consistency control on the data;

[0011] Step 4: Perform quality control based on historical extreme values ​​of the data;

[0012] Step 5: Verify the consistency of data within the system;

[0013] Step 6: Perform quality control on data of the same type;

[0014] Step 7: Perform quality identification on the data after quality control.

[0015] Furthermore, in step 1, the devices in the system include a laser wind measuring radar, a microwave radiometer, and a millimeter wave cloud radar; the data measured and recorded include temperature profile, relative humidity profile, wind speed, wind direction, and cloud phase data.

[0016] Furthermore, in step 2, after measuring and recording data for 24 hours, the temperature profile, relative humidity profile, wind speed, wind direction and cloud phase data are time-consistently controlled.

[0017] Furthermore, the time consistency control method is:

[0018] If N consecutive measurement data are the same within T hours, these N data are suspicious, except for missing data;

[0019] All data within one hour before and after the data being checked are used to form a sequence, and the mean and standard deviation of the sequence are calculated. If the absolute difference between the data being checked and the sequence mean is greater than or equal to three times the sequence standard deviation, the data being checked is considered suspicious.

[0020] Furthermore, in step 3, after measuring and recording the data for 24 hours, the temperature profile and relative humidity profile data are spatially consistent.

[0021] Furthermore, the method of spatial consistency control is:

[0022] The data of the detected pixel is compared with the average value of the data of the adjacent pixels. If it exceeds a given threshold, the data of the detected pixel is considered suspicious.

[0023] Furthermore, in step 4, historical data of more than 10 years in the area where the system is located is collected, and the historical extreme values ​​are obtained by processing. The data exceeding the extreme values ​​are determined as abnormal data and eliminated.

[0024] Furthermore, in step 5, the method for verifying the consistency of data within the system is: if the temperature T measured by the microwave radiometer is greater than or equal to the dew point temperature Td measured at the same time, the measured temperature T is reasonable; if the error between the temperature profile measured by the microwave radiometer and the temperature profile measured by the laser wind radar is less than △T, the measured temperature profile is reasonable; if the error between the humidity profile measured by the microwave radiometer and the humidity profile measured by the laser wind radar is less than △RH, the measured humidity profile is reasonable; if the error between the ground temperature measured by the microwave radiometer and the ground temperature measured by the laser wind radar is less than △Tg, the measured ground temperature is reasonable; if the error between the ground humidity measured by the microwave radiometer and the ground humidity measured by the laser wind radar is less than △RHg, the measured ground humidity is reasonable; if the error between the ground pressure measured by the microwave radiometer and the ground pressure measured by the laser wind radar is less than △Pg, the measured ground pressure is reasonable.

[0025] Furthermore, in step 6, based on the quality identification of two or more sets of data, combined with the analysis and evaluation of the accuracy of historical data of different devices, the status of the output data of different devices is judged, and data correction processing is carried out when necessary.

[0026] Furthermore, in step 7, the quality indicator includes: 0 indicates that the data is correct; 1 indicates that the data is suspicious; 2 indicates that the data is wrong; and 3 indicates that the data has not been quality controlled.

[0027] The beneficial effects of the present invention are:

[0028] The method disclosed in the present invention verifies the feasibility and effectiveness of controlling the data quality of the atmospheric vertical profile measurement system, can improve the quality of data products, and make the detection data of the atmospheric vertical profile measurement system more credible, thereby providing support for meteorological monitoring and forecasting, and realizing high-precision weather monitoring and forecasting. It is of great significance in the fields of aerospace measurement and control, and this method can also be extended to all atmospheric vertical profile comprehensive measurement systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic flow diagram of the method of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] Example 1: This example discloses a data quality control method for an atmospheric vertical profile measurement system. By further checking the data after quality control of a single device, performing time and space consistency quality control on the data content associated with different devices, quality control based on extreme values ​​of historical meteorological data, and consistency verification of meteorological elements within the system, the quality of data products can be further improved. Figure 1 As shown, the following steps are included:

[0032] Step 1: All devices in the system perform co-located synchronous observation, measurement and data recording;

[0033] The equipment in the system includes laser wind radar, microwave radiometer, and millimeter-wave cloud radar; the measured and recorded data include temperature profile (obtained by laser wind radar and microwave radiometer), relative humidity profile (obtained by laser wind radar and microwave radiometer), wind speed (obtained by laser wind radar), wind direction (obtained by laser wind radar), and cloud phase data (obtained by millimeter-wave cloud radar).

[0034] Step 2: After measuring and recording data for a certain period of time, perform time consistency control on the data;

[0035] After measuring and recording data for 24 hours, the temperature profile, relative humidity profile, wind speed, wind direction and cloud phase data are controlled for time consistency.

[0036] Time consistency control refers to the quality control of data that has a certain regularity within a certain time range. Most meteorological elements in this system (except wind direction, etc.) are continuously changing. Their changes over time should be continuous. At a certain time interval, the fluctuation of the same element should be within a certain range. The method of time consistency control is:

[0037] If N consecutive measurement data are the same within T hours, these N data are suspicious, except for missing measurements. The specific values ​​of T and N can be set separately according to different detection factors.

[0038] Test the limit value of the daily variation distribution of factors. For factors with certain daily variation patterns, such as temperature and relative humidity, the daily variation of the factor values ​​at each time has similar distribution characteristics. Use all data within 1 hour before and after the data being checked to form a sequence, calculate the mean and standard deviation of the sequence, and if the absolute difference between the data being checked and the sequence mean is greater than or equal to 3 times the sequence standard deviation, the data being checked is considered suspicious. Suppose the values ​​of a certain observation sequence are: X1, X2, X3...X n ;

[0039] Then the mean of the series is:

[0040] The standard deviation is:

[0041] Step 3: After measuring and recording data for a certain period of time, perform spatial consistency control on the data;

[0042] After measuring and recording data for 24 hours, the temperature profile and relative humidity profile data are spatially consistent.

[0043] The method of spatial consistency control is:

[0044] The data of the detected pixel is compared with the average value of the data of the adjacent pixels. If it exceeds the given threshold ΔX max , the data of the detected pixel is considered suspicious.

[0045] Assume that the observation value is: X i , the adjacent pixel observation value is: X i-1 、X i-2 、X i+1 、X i+2 , then the observed mean is:

[0046]

[0047] like: The observation is considered suspicious.

[0048] Step 4: Perform quality control based on historical extreme values ​​of the data;

[0049] Collect historical meteorological data for more than 10 years in the area where the system is located, process it to obtain historical extreme values ​​(minimum and maximum values), and determine the data that exceeds the extreme values ​​as abnormal data and eliminate them.

[0050] Quality control based on extreme values ​​of historical meteorological data is mainly used to identify whether the range of variation of data within a specified area and time domain is reasonable according to the elements. Data that exceeds the range of variation of elements is suspicious and should be further checked to determine whether the data is correct.

[0051] Check the boundaries of meteorological element measurements. The main measurement elements and measurement boundaries are:

[0052] The atmospheric temperature range is: -70℃~70℃;

[0053] The range of atmospheric relative humidity is: 0~100%;

[0054] The wind speed range is: -75m / s~75m / s;

[0055] The range of wind direction is: 0~360;

[0056] The intensity range is: -50dBZ~+35dBZ;

[0057] The speed measurement range is: -26m / s~+26m / s;

[0058] Spectral width range: 0~8m / s;

[0059] The range of linear depolarization ratio is: -35dB~5dB;

[0060] The range of differential reflectivity is: -3dB~+6dB;

[0061] The range of differential propagation phase shift rate is: -10~+10° / km;

[0062] The range of correlation coefficient is: 0 to 1.0.

[0063] Step 5: Verify the consistency of data within the system;

[0064] The consistency verification of meteorological element data within the system is mainly used to judge the rationality of the relationship between meteorological elements observed at the same time within the atmospheric vertical profile measurement system and to control data quality. The method for consistency verification of meteorological element data within the system is shown in the following table:

[0065] Verification of consistency of meteorological element data within the system

[0066]

[0067] Step 6: Perform quality control on data of the same type;

[0068] In cases where observations from different devices contain the same physical quantity (e.g., microwave radiometers and lidars both output temperature and relative humidity), two (or more) sets of data undergo a spatiotemporal consistency check and are considered suspicious if their absolute difference exceeds a certain range or their correlation coefficient (covariance) is less than a certain threshold. The quality control phase for associated parameters of the same type uses the quality identification of two or more sets of data, combined with an analysis and assessment of the accuracy of historical data from different devices, to determine the status of the output data from different devices and, if necessary, perform data correction.

[0069] Taking the temperature measurement by microwave radiometer and lidar as an example, suppose that the temperature sequence value observed by a microwave radiometer is: X1, X2, X3…X n , the temperature sequence values ​​observed by the lidar are: Y1, Y2, Y3…Y n , the data is considered qualified if it meets the following conditions:

[0070]

[0071] In the above formula, T min ,δ can be determined according to different user needs.

[0072] Step 7: Perform quality identification on the data after quality control.

[0073] Data quality identification is responsible for identifying the quality of data after it has undergone a unified quality control process, including integrity checks, limit value checks, range checks, spatial consistency checks, temporal consistency checks, and data quality identification. For data susceptible to weather conditions, quality control identification is added to suspicious data products during quality control. Quality control identification primarily provides data users with information about the quality of the data. Data quality identification includes: correct, suspicious, incorrect, and not quality controlled, represented by the following four codes:

[0074] 0: The data is correct; 1: The data is questionable; 2: The data is wrong; 3: The data has not been quality controlled.

Claims

1. A data quality control method for an atmospheric vertical profile measurement system, characterized in that: The steps include: Step 1: All devices in the system perform co-located synchronous observation, measurement and data recording; Step 2: After measuring and recording data for a certain period of time, perform time consistency control on the data; Step 3: After measuring and recording data for a certain period of time, perform spatial consistency control on the data; Step 4: Perform quality control based on historical extreme values ​​of the data; Step 5: Verify the consistency of data within the system; Step 6: Perform quality control on data of the same type; Step 7: Perform quality identification on the data after quality control.

2. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 1, the equipment in the system includes laser wind radar, microwave radiometer, and millimeter wave cloud radar; the data measured and recorded include temperature profile, relative humidity profile, wind speed, wind direction and cloud phase data.

3. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 2, after measuring and recording data for 24 hours, the temperature profile, relative humidity profile, wind speed, wind direction and cloud phase data are controlled for temporal consistency.

4. The data quality control method of the atmospheric vertical profile measurement system according to claim 3, characterized in that: The time consistency control method is: If N consecutive measurement data are the same within T hours, these N data are suspicious, except for missing data; All data within one hour before and after the data being checked are used to form a sequence, and the mean and standard deviation of the sequence are calculated. If the absolute difference between the data being checked and the sequence mean is greater than or equal to three times the sequence standard deviation, the data being checked is considered suspicious.

5. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 3, after measuring and recording data for 24 hours, the temperature profile and relative humidity profile data are spatially consistent.

6. The data quality control method of the atmospheric vertical profile measurement system according to claim 5, characterized in that: The method of spatial consistency control is: The data of the detected pixel is compared with the average value of the data of the adjacent pixels. If it exceeds a given threshold, the data of the detected pixel is considered suspicious.

7. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 4, historical data for more than 10 years in the area where the system is located is collected and processed to obtain historical extreme values. Data exceeding the extreme values ​​is determined as abnormal data and is eliminated.

8. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 5, the method for verifying the consistency of the data within the system is as follows: if the temperature T measured by the microwave radiometer is greater than or equal to the dew point temperature Td measured at the same time, the measured temperature T is reasonable; if the error between the temperature profile measured by the microwave radiometer and the temperature profile measured by the laser wind radar is less than △T, the measured temperature profile is reasonable; if the error between the humidity profile measured by the microwave radiometer and the humidity profile measured by the laser wind radar is less than △RH, the measured humidity profile is reasonable; if the error between the ground temperature measured by the microwave radiometer and the ground temperature measured by the laser wind radar is less than △Tg, the measured ground temperature is reasonable; if the error between the ground humidity measured by the microwave radiometer and the ground humidity measured by the laser wind radar is less than △RHg, the measured ground humidity is reasonable; if the error between the ground pressure measured by the microwave radiometer and the ground pressure measured by the laser wind radar is less than △Pg, the measured ground pressure is reasonable.

9. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 6, based on the quality identification of two or more sets of data, combined with the analysis and evaluation of the accuracy of historical data of different devices, the status of the output data of different devices is judged, and data correction processing is carried out when necessary.

10. The data quality control method of the atmospheric vertical profile measurement system according to claim 1, characterized in that: In step 7, the quality indicators include: 0 indicates that the data is correct; 1 indicates that the data is suspicious; 2 indicates that the data is wrong; and 3 indicates that the data has not been quality controlled.

Images