Method for carrying out quality control on hydro meteorological data by using relevance

Abstract

The invention discloses a method for performing quality control on hydrometeorological data by using relevance, and relates to the field of marine meteorology. According to the rule of hydro meteorological element data, data quality control is carried out by using the relevance between elements, namely the relevance quality control method. The method comprises the following steps: firstly, sorting data to enable the data to contain time, place, instrument state and other information, performing conventional quality control inspection and identifying abnormal values; the element relevance is analyzed and divided into natural relevance and algorithm relevance. According to natural association, data exception is judged by checking reasonability and contradiction of element values; according to algorithm association, after abnormity is detected in conventional quality control, it is judged that the associated derivative elements are abnormal. The correlation quality control covers a plurality of elements such as time, place, salinity, vector, relative humidity, wind-wave-flow and the like, and emphasizes that correlation is judged according to element observation sources and modes, so that excessive quality control is prevented. The method is clear in process and high in expansibility, suspicious or abnormal data can be effectively detected, and a reliable quality control method is provided for quality control of hydro meteorological observation data.

Description

Technical Field

[0001] The present invention relates to the field of marine meteorology, and particularly to a method for quality control of hydrometeorological data using relevance. Background Art

[0002] Marine environmental observation data provides important basic information and is an indispensable basis for aspects such as marine environmental forecasting, marine engineering construction, response to marine disasters, and protection of marine rights and interests. In particular, high-quality observation data can scientifically reflect the marine environment. However, during the actual observation process, a series of factors such as data collection, transmission, storage, instrument failures, and accidental events at sampling locations may affect the observation results, leading to abnormal data. Implementing accurate and effective data quality control for these data is one of the major challenges faced by marine workers.

[0003] Marine monitoring data has characteristics such as multi-source, multi-state, diversity, and regionality, which determines that the control and evaluation of the quality of marine observation data cannot be generalized and need to be comprehensively considered in combination with specific observation methods, observation platforms, and observation regions. Although there have been a large number of research results in the field of marine data quality control by predecessors, and the methods and processes of quality control are very diverse, during the process of marine data quality control, many obvious incorrect data still cannot be effectively detected, or some observation data are misjudged as abnormal data because of large changes, while in fact the mutations of these data are true descriptions of environmental changes. Based on the quality control of multi-platform and long-term hydrometeorological observation data, by sorting out and analyzing these undetected or misdeleted data, it is found that some abnormal data that are difficult to judge using conventional quality control methods are more or less related to the changes of other elements to a certain extent. In view of this, the present invention establishes a relevance-based quality control method. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for quality control of hydrometeorological data using relevance, aiming to solve the problem that traditional quality control methods are difficult to effectively detect some abnormal data or misjudge normal data, and use the relevance between elements for quality control to obtain complete, accurate, and reliable hydrometeorological observation data.

[0005] To achieve the above-mentioned invention purpose, the present invention provides the following technical solutions.

[0006] A relevance-based quality control method for hydrometeorological data quality control includes:

[0007] (1) Data sorting step: Sort the hydrometeorological data, which should include basic information such as time, location, and instrument status;

[0008] (2) Routine quality control steps: Conduct routine quality control inspections on all imported data, such as threshold tests, Leite tests (three times the standard deviation), Neill tests, Grubbs tests, Dixon tests, kurtosis tests, gradient tests, spike tests, dead value tests (also known as card value tests), visual manual inspections, etc., and identify outliers;

[0009] (3) Association analysis steps: Analyze the possible associations between different elements, which are divided into natural associations and algorithmic associations according to the association methods;

[0010] (4) Natural association quality control steps: Examine whether the values between various elements are reasonable or contradictory to determine whether the observed element data is abnormal. If the test fails, all elements with natural associations are determined to be suspicious or abnormal; for example, the associations between basic information such as time, location, and instrument working status and various elements, the associations between wind, wave, and current, etc.;

[0011] (5) Algorithmic association quality control steps: After detecting abnormal data using routine quality control methods, all associated derived elements are judged to be suspicious or abnormal; for example, the associations between temperature, conductivity, pressure, and salinity, the associations between the magnitude and direction of vectors, etc.; that is, if an abnormality is found in the temperature during quality control, the corresponding salinity is considered abnormal, but an abnormal salinity cannot determine the corresponding temperature abnormality because it may be due to an abnormal conductivity or an abnormality in the salinity itself;

[0012] (6) Re - quality control steps: Optionally, after using association - based quality control, if necessary, the result data can be re - quality - controlled using conventional methods.

[0013] The association - based quality control includes various elements such as time - related association quality control, location - related association quality control, salinity - related association quality control, vector - related association quality control, relative humidity - related association quality control, wind - wave - current - related association quality control, etc. The association is judged based on the element observation sources and methods to prevent over - quality control.

[0014] In time - related association quality control, a threshold test is conducted on time. The month is limited between 1 and 12, the date is set within a reasonable range according to different months (the maximum for April, June, September, and November is 30 days; in a leap year, February has 29 days, and in a common year, it has 28 days; the maximum for the other months is 31 days), the hour is between 1 and 24, and the minute and second are between 0 and 60. At the same time, monotonicity and continuity tests are conducted; if there is solar radiation data, the data is judged based on the correspondence between sunrise and sunset times and solar radiation intensity. Data with solar radiation intensity significantly higher than 0 after sunset or close to 0 during the day is determined to be abnormal data, and the corresponding hydro - meteorological observation data at that time point is determined to be suspicious or abnormal data.

[0015] In the quality control of location correlation, after converting the longitude and latitude recording methods of the observation data locations into consistent units, one or more of the following tests are performed: threshold test, landing test, three - standard - deviation test, Grubbs test, and Dixon test. For different observation carriers, it is judged whether the observation carrier is abnormal according to the variation law of longitude and latitude with time. If the observation carrier is a ship, fixed - point or underway observations are considered. If it is a surface - drifting buoy, it should move with the ocean current. A fixed - point observation buoy normally swings with the tide only in a specific area. If abnormal movement is found, such as the case of the No. 4 large buoy of Fujian Forecast Station running off - station, all the data corresponding to the running - off - station period are judged as suspicious or abnormal. For the longitude and latitude data where abnormal values are detected during quality control and cannot be corrected, the hydrometeorological observation data at this point are marked as suspicious or incorrect.

[0016] In the quality control of salinity correlation, since seawater salinity is a function of parameters such as temperature, pressure, and conductivity, one or more of the following tests are performed on temperature and conductivity for quality control: blank - value test, time test, location test, equipment - log test, threshold test, three - standard - deviation test, kurtosis test, Grubbs test, Dixon test, gradient test, frozen - value test, and manual visualization test. Based on the quality - control results of temperature and conductivity, the salinity is subjected to correlation - based quality control. As long as one of the parameters of temperature, pressure, and conductivity is abnormal, the corresponding salinity is judged as abnormal.

[0017] In the quality control of vector correlation, one or more of the following tests are performed on the vector magnitude: threshold test, three - standard - deviation test, Grubbs test, Dixon test, gradient test, frozen - value test, and manual visualization test. One or more of the following tests are performed on the direction: threshold test and frozen - value test. If one of the tests for magnitude and direction fails, the vector is judged as suspicious or abnormal. The tested vector is decomposed into an east component and a north component, and then one or more of the following tests are performed on the two components: three - standard - deviation test, Grubbs test, Dixon test, gradient test, and frozen - value test. If one of the two components fails the test, the vector is judged as suspicious or abnormal.

[0018] In the quality control of relative - humidity correlation, one or more of the following tests are performed on the atmospheric temperature: time test, location test, equipment - log test, threshold test, three - standard - deviation test, kurtosis test, Grubbs test, Dixon test, gradient test, and frozen - value test. If an abnormal value is found in the atmospheric temperature during the test, the corresponding relative humidity is also judged as an abnormal value.

[0019] In the quality control of the wind-wave-current correlation, considering the complex non-linear interaction among wind, wave and current, when the change range of wind is large, sudden changes in waves and current are allowed; according to the corresponding relationship between wind speed and wave height specified in the national standard HY / T 0315—2021, combined with the curve distribution of wave height and maximum wind speed, judge whether the wave height data is abnormal, and avoid misjudging the maximum value of wave height under special weather such as typhoon as an abnormal value.

[0020] In addition, quality control of correlation is also carried out on other elements that may have a large correlation, including temperature and dew point, weather phenomenon and visibility, maximum wave height and significant wave height, maximum wave height and mean wave height, flow velocity of underway observation and attitude data of the ship, flow velocity of mooring observation and attitude data of the instrument itself, etc. According to the correlation characteristics among the elements, corresponding quality control inspection methods are adopted for quality control.

[0021] During the quality control process, fully understand the observation sources and observation methods of each element, and then judge whether there is a real correlation between the elements, so as to prevent over-quality control of elements without correlation.

[0022] Compared with the prior art, the technical effects and advantages of the present invention are as follows:

[0023] 1. Improve data accuracy: By exploring the correlation among each element, abnormal data that is easily missed by conventional quality control methods can be accurately identified; for example, in the quality control of salinity correlation, with the help of the quality control results of temperature and conductivity, salinity abnormal values can be effectively identified, greatly improving the accuracy of data and laying a solid foundation for subsequent analysis, research and application based on these data.

[0024] 2. Avoid misjudgment and misdeletion: For some situations where data mutations occur due to environmental changes, traditional methods are prone to misjudge as abnormal and delete, while the correlation quality control method of the present invention fully considers the internal relationship among elements; taking the wind-wave-current correlation as an example, when the wind changes greatly, it can reasonably judge that the sudden changes in waves and current are normal, and avoid misdeleting the data that truly reflects environmental changes, ensuring the integrity of the data.

[0025] 3. Wide applicability: Correlation quality control covers a variety of elements, from time, location to temperature, salinity, vector, etc., and is applicable to various hydrological and meteorological data scenarios. Whether it is ocean monitoring buoy data or research ship data, this method can be used for effective quality control and can be widely applied to many fields such as ocean environment forecasting, ocean engineering construction, and ocean disaster response.

[0026] 4. Clear and easy-to-operate process: The method steps are clear. First, the data is sorted out, then conventional quality control is carried out, and then the correlation of elements is analyzed and quality control is carried out accordingly. The process logic is clear and convenient for ocean workers to understand and operate. Even beginners can control the quality of hydrological and meteorological data step by step according to the process.

[0027] 5. Strong scalability: With the in-depth study of the correlation between hydrometeorological elements, newly discovered correlation relationships can be continuously incorporated into the method. For example, if new element correlations are discovered subsequently, they can be easily added to the existing correlation-based quality control system to continuously optimize the data quality control effect and meet the ever-changing hydrometeorological data processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic flowchart of an embodiment of the present invention.

[0029] Figure 2 It is the solar short-wave radiation of the Jia Geng on May 29, 2017.

[0030] Figure 3 It is the moving track of the No. 4 large buoy of the Fujian Forecast Station from April 29 to 30, 2017.

[0031] Figure 4 It is a comparison chart of the surface water temperature quality control of the Strait 2 buoy from December 16 to 31, 2017 before and after.

[0032] Figure 5 It is a comparison chart of the surface salinity quality control of the Strait 2 buoy from December 16 to 31, 2017 before and after.

[0033] Figure 6 It is a chart of the wind speed magnitudes of 5 buoys in the waters near Zhangzhou from March 1 to 5, 2013.

[0034] Figure 7 It is the wind speed vector diagram of 5 buoys in the waters near Zhangzhou before quality control from March 1 to 5, 2013.

[0035] Figure 8 It is the wind speed vector diagram of 5 buoys in the waters near Zhangzhou after quality control from March 1 to 5, 2013.

[0036] Figure 9 It is a comparison chart of the air temperature quality control of the No. 1 large buoy of the Forecast Station from August 1 to 15, 2017 before and after.

[0037] Figure 10 It is a comparison chart of the relative humidity quality control of the No. 1 large buoy of the Forecast Station from August 1 to 15, 2017 before and after.

[0038] Figure 11 It is the maximum wave height of the Beishuang buoy from July 16 to 31, 2017.

[0039] Figure 12 It is the maximum wind speed of the Beishuang buoy from July 16 to 31, 2017. DETAILED DESCRIPTION OF THE INVENTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not used to limit the present invention. On the contrary, the present invention covers any alternatives, modifications, equivalent methods and solutions made within the spirit and scope of the present invention as defined by the claims.

[0041] In view of the laws of hydrometeorological element data, the present invention uses the correlation between elements for data quality control, that is, the correlation-based quality control method. Correlations widely exist among various elements. In addition to the correlations between basic information such as time, location, and instrument status and various elements, there are also correlations between elements. For example, there are correlations between wind, wave and current, temperature, salinity, pressure and salinity, temperature and relative humidity, wind speed and wind direction, flow velocity and flow direction, temperature and dew point, weather phenomenon and visibility, maximum wave height and significant wave height, maximum wave height and average wave height, flow velocity measured by underway observation and the attitude data of the ship, etc.

[0042] As Figure 1 shown, the embodiments of the present invention include the following steps:

[0043] 1. Organize hydrometeorological data:

[0044] First, organize the hydrometeorological data. Data from different observation platforms can be widely collected, such as ocean monitoring buoys, scientific research vessels, meteorological stations, etc. Ensure that the data covers basic information such as time, location, and instrument status. For time information, it is necessary to clarify its recording format, such as whether it is in the standard year-month-day hour:minute:second format. If there are time zone differences, they need to be uniformly marked. When the location information is represented by longitude and latitude, it is necessary to check whether it is in the standard degree, minute, second or decimal degree representation. If there are different unit representations, they need to be converted and unified. Instrument status information should record in detail the instrument model, calibration time, last maintenance time, etc. to judge the reliability of the instrument. Store the collected data according to a certain database structure for convenient subsequent calling and analysis. For data from different sources, integration processing is required to ensure the consistency and integrity of the data. For example, compare the data of the same element measured by different instruments at the same time and location, and remove obviously incorrect or duplicate data.

[0045] 2. Conduct routine quality control inspections on all imported data:

[0046] Flexibly select appropriate conventional quality control inspection methods according to the data characteristics and element types. For numerical data, such as temperature and wind speed, threshold inspection can be preferentially adopted, setting reasonable upper and lower threshold values, and the data outside the threshold range is marked as abnormal. For data suspected of having outliers, methods such as Grubbs test and Dixon test can be used for detection. For the data distribution characteristics, the kurtosis test can be used to judge whether it conforms to the normal distribution characteristics. For time series data, methods such as gradient test and spike test can be used to detect whether the change trend of the data is reasonable. After various inspections, the data marked as abnormal should be recorded in detail, including the specific values of the abnormal data, the time points, the elements involved, and other information. At the same time, an outlier database should be established to facilitate the subsequent analysis and processing of abnormal data.

[0047] 3. Analyze the possible correlations between different elements:

[0048] Collect and sort out the known correlation knowledge between various elements in the hydrometeorological field, such as the relationships between temperature and relative humidity, salinity and temperature, conductivity, etc., and establish a correlation knowledge base. The knowledge base can be stored in forms such as tables and graphs for convenient query and update. Use data analysis techniques to mine and analyze a large amount of historical hydrometeorological data to discover potential element correlations. For example, through methods such as correlation analysis and clustering analysis, find the strong correlation relationships existing in the data. For the newly discovered correlations, verification and evaluation are required to ensure their reliability. According to the correlation methods of different ocean elements, the correlation quality control methods are divided into algorithm correlation quality control and natural correlation quality control.

[0049] 4. For natural correlations, check whether the values between various elements are reasonable or contradictory to determine whether the observed element data is abnormal:

[0050] According to the correlation knowledge base and actual business requirements, formulate detailed natural correlation inspection rules. For example, for the correlation between time and solar radiation, it is stipulated that the solar radiation intensity during the day should be greater than a certain threshold, and close to zero at night. For the correlations among wind, wave, and current, according to different wind speed ranges, set reasonable ranges for the corresponding wave height and flow velocity. According to the formulated inspection rules, check the data one by one. For the data that does not conform to the rules, it is judged as suspicious or abnormal. At the same time, record the detailed information of the abnormal situation, such as which element values are unreasonable or contradictory, for subsequent cause analysis.

[0051] 5. For algorithm correlations, after using conventional quality control methods to detect abnormal data, all the derived elements associated with it are judged as suspicious or abnormal:

[0052] First, clarify the algorithmic correlation relationships among various elements, such as the functional relationships between salinity and temperature, conductivity, and pressure. For complex algorithmic correlations, mathematical models or machine learning models can be used for description. After detecting abnormal data of a certain element through conventional quality control methods, judge whether the related derivative elements are abnormal according to the algorithmic correlation relationships. For example, if the temperature is detected to be abnormal, judge whether the corresponding salinity is abnormal according to the correlation algorithm between salinity and temperature. For the derivative elements judged to be abnormal, make detailed records and annotations.

[0053] 6. Optionally, perform conventional method quality control on the result data after correlation-based quality control again:

[0054] According to the results and data characteristics of the first conventional quality control, select a suitable conventional quality control method for re-inspection. For example, for the data that did not undergo dead value inspection in the first conventional quality control, dead value inspection can be carried out in this inspection. For the element data that showed abnormalities in the correlation-based quality control, visual manual inspection can be focused on, and it can be judged whether the data is really abnormal through manual observation. According to the results of the second conventional quality control, verify and correct the data. For the data still judged to be abnormal, the reasons can be further analyzed, such as whether it is an instrument failure, data transmission error, etc. For the incorrect data, it can be corrected or deleted according to the actual situation.

[0055] Through the use of correlation-based quality control for hydrometeorological observation elements with correlations, the present invention can more effectively detect suspicious or abnormal data, providing a reliable quality control method for the quality control of hydrometeorological observation data.

[0056] The data used in the following embodiments are from the marine environment monitoring buoys of the Fujian Ocean Forecast Station and the Jiageng research vessel of Xiamen University. The buoy data of the forecast station spans from January 1, 2017 to December 31, 2022. There are 8 large buoys and 40 small buoys, mainly including environmental factors such as flow velocity, flow direction, wind speed, wind direction, wave height, wave direction, wave height period, wave number (i.e., number of measured waves), water temperature, conductivity, salinity, air temperature, air pressure, visibility, relative humidity, dew point, and basic information such as time, latitude, longitude, and instrument status. The observation factors of different buoys are not exactly the same. The Jiageng research vessel data spans from May 26, 2017 to December 27, 2022, including environmental factors such as air temperature, air pressure, visibility, relative humidity, dew point, rainfall, solar shortwave radiation, and basic information such as time, latitude, longitude, and instrument status. The data of Strait No. 2 buoy include: significant wave height (m), significant wave period (s), average wave height (m), average period (S), average wave direction (°), maximum wave height (m), maximum wave height period (s), 1 / 10 large wave height (m), 1 / 10 large wave period (s), air temperature (℃), air pressure (hPa), visibility (km), relative humidity (%), surface water temperature (℃), surface salinity, longitude (°E), latitude (°N), etc.

[0057] The hydrological and meteorological observation data are subjected to correlation quality control inspection, and the processing flow is as follows:

[0058] 1) Time-related quality control

[0059] In actual observation, there are many reasons for abnormal time data. The most common ones are recording errors or omissions. It may also be that the time setting of the observation instrument is wrong, the initial time of the new instrument is not set correctly, or the internal clock module of the instrument is faulty, which will cause the recorded time to deviate; and time confusion and abnormal time data caused by inconsistent time zones (such as East 8 and Universal Time) or different time systems (such as 12 hours and 24 hours).

[0060] When performing quality control on time correlation, in order to ensure that the time data is within a reasonable range, a threshold test is performed on the time. The month is between 1 and 12, the date is between 1 and 31 (April, June, September, and November have a maximum of 30 days; February has 29 days in a leap year and 28 days in a common year; the maximum number of days in other months is 31 days), the hour is between 1 and 24, and the minute and second are between 0 and 60. Any data exceeding these threshold ranges is initially judged to be abnormal. In addition to the threshold test, monotonicity and continuity tests are performed. Monotonicity requires that the time data increases in sequence with the observation order, and time cannot flow backwards. This is the basis for ensuring the correctness of the time logic. The continuity test ensures that the time intervals are uniform, there are no time jumps or interruptions, and the coherence of the time series is maintained.

[0061] Auxiliary data association quality control: If there is solar radiation data, quality control is carried out according to the correspondence between sunrise and sunset times and solar radiation intensity. The solar radiation intensity during the day should be higher than at night, and the solar radiation intensity should be close to 0 after sunset. Taking the solar shortwave radiation of the Jia Geng scientific research ship as an example, see Figure 2 , in the figure, the solar radiation during the day is 0, and the maximum peak appears at night. This may be due to an incorrect time setting, so it is judged as abnormal data. If it is determined as a time error through quality control, the corresponding hydrometeorological observation data at this time point can be judged as suspicious or abnormal data. Once it is determined through the above test that there is an error in the time data, all hydrometeorological observation data corresponding to this time point, whether it is basic meteorological elements such as temperature, wind speed, and air pressure, or ocean elements data such as waves and ocean currents, are judged as suspicious or abnormal data. It cannot be directly used for subsequent analysis and research.

[0062] 2) Location association property quality control

[0063] The location of the observation data is generally represented by a set of longitude and latitude. In actual observations, in addition to abnormalities such as incorrect or missing records of longitude and latitude, another main reason for the confusion is the difference in units. The longitude and latitude of the location may be expressed in degrees, degrees and minutes, degrees, minutes and seconds, or the unit may be omitted. This inconsistency easily leads to errors in data processing and analysis. Therefore, before quality control, various recording methods should be converted to the same unit before testing.

[0064] For different observation carriers, the laws of longitude and latitude changes are also different. The abnormality of the observation carrier can be judged by the change of longitude and latitude over time. If the observation carrier is a ship, it is divided into fixed-point or underway observations. During fixed-point observations, the ship's position is relatively fixed, and the longitude and latitude basically remain unchanged; during underway observations, the longitude and latitude will change continuously along the sailing route. If it is a surface drifting buoy, it depends on the ocean current drive and will continuously move with the ocean current, and its longitude and latitude will change continuously. A fixed-point observation buoy normally does not move and only swings within a specific area with the tide. If it is found to move, a buoy running event may occur. The running track of the No. 4 large buoy of the Fujian Forecast Station is shown in Figure 3, during the running bid period, all corresponding data are judged as suspicious or abnormal. When conducting quality control on latitude and longitude data, methods such as threshold test, landing test, three - standard - deviation test, Grubbs test, and Dixon test can be used. The threshold test determines a reasonable range for latitude and longitude. For example, the latitude ranges from - 90° to 90°, and the longitude ranges from - 180° to 180°. Data outside this range are marked as suspicious. The landing test is used to determine whether the latitude and longitude correspond to the actual geographical location, avoiding unreasonable situations such as data falling in the ocean but being recorded as a land location. The three - standard - deviation test, Grubbs test, and Dixon test focus on identifying outliers in latitude and longitude data, that is, data points with a large difference from other data. For data that is found to be abnormal during quality control and cannot be corrected, the hydrometeorological observation data at this point is marked as suspicious or incorrect. If it is found through the above tests that the latitude and longitude data are abnormal and cannot be corrected, then all hydrometeorological observation data corresponding to this latitude and longitude, such as temperature, wind speed, humidity, etc., should be marked as suspicious or incorrect data. Through quality control based on location association, abnormal data caused by incorrect location information can be effectively screened out, providing a more accurate data basis for hydrometeorological research and applications.

[0065] 3) Quality control based on salinity association

[0066] The salinity of seawater is not easily measured directly and is generally calculated from other quantities through specific formulas. According to the expression of the seawater state equation TEOS - 10 introduced by the United Nations Educational, Scientific and Cultural Organization in 2010, the calculation of salinity has been revised from the previously used practical salinity to more accurate absolute salinity, but it is still essentially a function of parameters such as temperature, pressure, and conductivity. Since salinity is a function of conductivity, temperature, and depth, if one of these parameters is abnormal, the corresponding salinity is determined to be abnormal. This way of determining abnormalities based on parameter association is the core logic of quality control based on salinity association.

[0067] In the nearshore area, due to the influence of various factors such as the freshwater injection at the river estuary and tidal changes, the salinity change is relatively complex and has a large amplitude, making it difficult to perform quality control. However, the measurements of temperature and conductivity are relatively stable and are relatively easy to perform quality control on. Methods such as blank value test, time test, location test, equipment log test, threshold test, three - standard - deviation test, kurtosis test, Grubbs test, Dixon test, gradient test, stagnant value test, and manual visualization test can be used to perform quality control on temperature and conductivity. For the blank value test, check whether the measured value is the default filled value; for the time test, analyze the variation law of the data in the time series and check for any abnormal fluctuations; for the location test, based on the marine environmental characteristics of different geographical locations, judge whether the measured value at this location is reasonable; for the equipment log test, confirm whether the instrument is working properly by referring to the operation records of the measuring equipment; for the threshold test, set a reasonable numerical range, and the data outside the range is considered abnormal; for the three - standard - deviation test, identify the data that deviates too much from the mean value; for the kurtosis test, judge whether the data distribution conforms to the characteristics of a normal distribution; for the Grubbs test and Dixon test, they are used to detect outliers in the data; for the gradient test, check whether the change gradient of the data over time or space is normal; for the stagnant value test, check whether the data remains unchanged for a long time, and for the manual visualization test, discover obvious abnormal points by directly observing the data or plotting charts.

[0068] The quality control results of water temperature are shown in Figure 4 , and the salinity is subjected to associated quality control using the quality control results of temperature and conductivity. The results of the salinity associated quality control are shown in Figure 5 . It can be seen from the figure that all the abnormal data of salinity are detected. From Figure 4 's water temperature quality control results and Figure 5 's salinity associated quality control results, it can be intuitively seen that by performing quality control on temperature and conductivity and using this as a basis to perform associated quality control on salinity, the abnormal data of salinity are successfully detected. It shows that the method of the present invention can effectively identify the salinity anomalies caused by abnormal related parameters.

[0069] 4) Vector associated quality control

[0070] A vector is divided into magnitude and direction. For the magnitude of the vector, threshold test, three - standard - deviation test, Grubbs test, Dixon test, gradient test, stagnant - value test and manual visualization test are adopted, etc. The threshold test sets a reasonable range of magnitudes, and values outside the range are concerned; the three - standard - deviation test can identify data that deviates too much from the mean; the Grubbs and Dixon tests help to find out possible outliers; the gradient test checks whether the data change trend is reasonable; the stagnant - value test checks whether the data remains unchanged for a long time, and marks it if there is an abnormality. For the direction of the vector, threshold test and stagnant - value test are used. A reasonable interval of the direction is set. If the direction exceeds the threshold, or remains fixed for a long time and does not conform to the actual situation, it is determined to be abnormal. If either the magnitude or the direction fails a test, the vector is determined to be suspicious or abnormal.

[0071] The vectors after preliminary testing are decomposed into east - component and north - component, and then quality control is carried out. For the two components, three - standard - deviation test, Grubbs test, Dixon test, gradient test and stagnant - value test are adopted, etc. If either of the two components of the east - component or the north - component fails a test, even if the preliminary tests of the magnitude and direction seemed normal before, the vector is determined to be suspicious or abnormal.

[0072] Taking the synchronous observations of 5 buoys near Zhangzhou as an example, the wind - speed magnitudes are shown in Figure 6 , the wind - speed vectors are shown in Figure 7 , and the wind - speed vectors after associated quality control are shown in Figure 8 . It can be seen from the figure that although the wind - speed magnitudes are normal, the data with abnormal wind directions are all excluded. From the Figure 6 wind - speed magnitude data, the values are all within the normal range. From the Figure 7 wind - speed vector diagram, it is found that some wind directions are abnormal. Through the vector - associated quality - control process, the wind - speed vectors are decomposed into east - component and north - component for in - depth testing. Finally, in the Figure 8 wind - speed vector diagram after associated quality control, the data with abnormal wind directions are successfully excluded. Experiments show that vector - associated quality control can effectively identify abnormal situations that are difficult to detect only from the wind - speed magnitude, and comprehensively improve the data quality.

[0073] 5) Relative - humidity associated quality control

[0074] Relative humidity is related to two quantities. One is the actual water vapor density in the air, and the other is the saturated water vapor density at the same temperature. The saturated water vapor density is generally obtained by looking up tables. Therefore, abnormal atmospheric temperature will definitely lead to abnormal relative humidity. When conducting relative humidity associated property quality control, time inspection, location inspection, equipment log inspection, threshold inspection, three - standard - deviation inspection, kurtosis inspection, Grubbs inspection, Dixon inspection, gradient inspection, and stagnant value inspection are carried out on the atmospheric temperature. Time inspection is used to verify the rationality of temperature data in the time series, ensuring that its change over time conforms to natural laws and there is no abnormal fluctuation caused by time disorder. Location inspection determines whether the temperature at a location is within a reasonable range based on the climate characteristics of different geographical locations. Equipment log inspection checks the log records of the measuring equipment to confirm whether the instrument is operating normally and whether there are temperature measurement deviations caused by faults. Threshold inspection sets reasonable upper and lower limits for temperature, and data outside the range is immediately marked. Three - standard - deviation inspection identifies abnormal temperature values that deviate too much from the mean. Kurtosis inspection is used to determine whether the distribution of temperature data conforms to the characteristics of a normal distribution. If there is a significant deviation, it may mean that there are abnormal data. Grubbs inspection and Dixon inspection focus on detecting outliers in the data. Gradient inspection checks whether the change gradient of temperature over time or space is reasonable, and stagnant value inspection focuses on whether the temperature remains unchanged for a long time. If abnormal situations occur, the corresponding data will be key - checked. When abnormal values of atmospheric temperature are determined through the above - mentioned multiple inspection methods, based on the close relationship between relative humidity and temperature, without the need for a separate and complex inspection process for relative humidity, the corresponding relative humidity can be directly determined as an abnormal value. The temperature quality control results are shown in Figure 9 , and the corresponding relative humidity associated property quality control results are shown in Figure 10 . From Figure 9 , it can be seen that the temperature did not change for several days and was the same value. This is typical of stagnant values in instrument measurement. These unchanging values are all determined as abnormal values. From Figure 10 , it can be seen that relative humidity is highly correlated with temperature. When the temperature is abnormal, the relative humidity in the corresponding part is also abnormal. Therefore, through associated property quality control, the relative humidity data corresponding to abnormal temperature can be directly determined as abnormal data.

[0075] 6) Wind, wave, and current associated property quality control

[0076] There are very complex non - linear interactions among wind, waves, and currents. Wind is the initial driving force. When persistent wind acts on the sea surface, the seawater will flow, forming wind - driven currents. At the same time, when the wind blows onto the sea surface, it frictions with the seawater, triggering water body fluctuations and forming wind waves. After the wind waves leave the wind field, they will turn into swells. As the wind field intensifies and time goes on, both wind waves and swells may become huge waves. Regarding the correlation among wind, waves, and currents in the ocean, it is difficult to directly describe with parameters and requires statistics. However, one thing is relatively certain. Wind changes are relatively easy to predict or observe. When the wind changes significantly, sudden changes in waves and currents should be regarded as normal phenomena and should not be excluded during quality control.

[0077] Taking the observation data of the Beishuang small buoy as an example, the maximum wave height suddenly increased from July 29th to 30th, 2017. See Figure 11 , and from Figure 11 it can be seen that the maximum wave height suddenly increased; through analysis, it was due to Typhoon Haitang No. 1710 landing in Fuqing, Fujian on the evening of July 30th. Combining Figure 12 with the maximum wind speed data measured by the Beishuang small buoy in , it can be known that this change in wave height was caused by the strong wind brought by the typhoon. When conducting quality control on the wave height data, if only looking at the wave height data in isolation and according to the conventional quality control standards, it is very easy to misjudge the maximum value of the wave height on typhoon days as an abnormal value. It can be seen from the national standard HY / T0315 - 2021 that when the wind speed is greater than 30 m / s, the reasonable distribution range of the wave height is 2.5 - 17.0 m. Combining with the curve distribution of the wave height and the maximum wind speed, it can be seen that the maximum value of the wave height slightly lags behind the maximum value of the maximum wind speed in time, which conforms to the physical law of the interaction among wind, waves, and currents. Therefore, the maximum value of the wave height corresponding to this wind speed is the data that truly reflects the ocean state under typhoon weather and belongs to the correct value and should not be deleted.

[0078] 7) Quality control for the correlation of other elements

[0079] In addition, there may also be elements with greater correlations, such as temperature and dew point, weather phenomena and visibility, maximum wave height and significant wave height, maximum wave height and mean wave height, the flow velocity measured during underway observations and the attitude data of the ship, the flow velocity measured by moored observations and the attitude data of the instrument itself, etc.

[0080] 8) Precautions for correlation - based quality control

[0081] For correlation - based quality control of hydrometeorological data, it is necessary to fully understand the observation sources and observation methods of these elements, and then judge whether there is a real correlation between the elements to prevent excessive quality control of elements without correlation.

[0082] From the above several elements, the effect of correlation-based quality control is better. It can not only judge whether the observed elements are suspicious or abnormal through information such as time and location, but also determine whether the observed elements are abnormal through the correlation between various elements. When using the algorithm-based correlation quality control, it is necessary to cooperate with other quality control methods. After detecting abnormal data of a certain element using other quality control methods, the derived elements associated with it can be judged as abnormal. When using natural correlation quality control, it is mainly to check whether the values between various elements are reasonable or contradictory. However, it should be noted that the correlation-based quality control method is only applicable to elements with a correlation, and it cannot be used arbitrarily to avoid over-quality control, and strive to effectively detect abnormal data without misdeleting correct data.

[0083] From the results of correlation-based quality control, the present invention establishes a scientific and practical data quality control method for hydrometeorological observation data, providing guarantee for finally obtaining complete, accurate and reliable-quality hydrometeorological observation data, so as to serve aspects such as marine environmental forecasting, marine engineering construction, marine disaster response, marine rights and interests protection, marine economic construction, and marine emergency management.

[0084] The present invention innovatively proposes a method for quality control of hydrometeorological data by using the correlation between elements. Traditional quality control methods mostly test single elements in isolation, while this method breaks through this limitation. Considering the complex internal relationships between hydrometeorological elements, by analyzing natural correlation and algorithmic correlation, it can more comprehensively and deeply detect abnormal situations in the data, which is a novel and effective strategy in the field of data quality control. The present invention incorporates various elements such as time, location, salinity, vector, relative humidity, wind-wave-current, etc. into the correlation-based quality control system, formulates specialized quality control methods for the characteristics of different elements, and forms a multi-dimensional and all-round quality control framework. This quality control method that comprehensively considers the correlation of multiple elements has significant innovation and advancement compared with traditional quality control methods for single elements or simple combinations. The present invention effectively solves the problems of easy omission of abnormal data and misjudgment and misdeletion of real data in traditional quality control methods. It improves the data quality and meets the requirements for the accuracy and integrity of hydrometeorological data in practical applications. The correlation-based hydrometeorological data quality control method established by the present invention can effectively control the quality of hydrometeorological data, and can better detect some suspicious data or abnormal data that are difficult to detect by conventional methods. This method has a clear process, is easy to learn and use, and has strong scalability, providing a reliable quality control method for the quality control of the entire hydrometeorological data.

[0085] The above embodiments are only preferred embodiments of the present invention and cannot be considered as limiting the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of the application of the present invention shall still fall within the scope covered by the patent of the present invention.

Claims

1. A method for quality control of hydrological and meteorological data using correlation, characterized in that: The following steps are involved: 1) Data collation step: collate the hydrological and meteorological data to include basic information such as time, location, and instrument status; 2) Routine quality control steps: Perform routine quality control tests on all imported data and identify outliers. Routine quality control tests include one or more of the following: threshold test, Wright test, Nair test, Grubbs test, Dixon test, kurtosis test, gradient test, spike test, stiffness test, and visual manual test; 3) Correlation analysis step: Analyze the possible correlation between different elements, which can be divided into natural correlation and algorithmic correlation according to the correlation mode; 4) Natural association quality control step: By checking whether the values ​​of each factor are reasonable or contradictory, determine whether the observed factor data is abnormal. If the test fails, the naturally associated factors are judged as suspicious or abnormal; 5) Algorithm-related quality control step: After abnormal data is detected using conventional quality control, the derivative factors associated with it are judged as suspicious or abnormal.

2. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: Correlation quality control includes time-correlated quality control, location-correlated quality control, salinity-correlated quality control, vector-correlated quality control, relative humidity-correlated quality control, wind-wave-current-correlated quality control and many other factors. The correlation is judged based on the source and method of factor observation to prevent excessive quality control.

3. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: In the time-related quality control, a threshold test is performed on the time, and the month is limited to between 1 and 12. The date is set in a reasonable range according to different months: April, June, September, and November have a maximum of 30 days; February has 29 days in a leap year and 28 days in a common year; the other months have a maximum of 31 days, the hour is between 1 and 24, and the minute and second are between 0 and 60; monotonicity and continuity tests are performed at the same time; if there is solar radiation data, it is judged based on the corresponding relationship between sunrise and sunset time and solar radiation intensity, and the data with solar radiation intensity significantly higher than 0 after sunset or close to 0 during the day are judged as abnormal data, and the hydrological and meteorological observation data corresponding to the time point are judged as suspicious or abnormal data.

4. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: In the location-related quality control, the longitude and latitude recording methods of the observation data locations are converted into consistent units and then tested, using one or more of the threshold test, landing test, three times standard deviation test, Grubbs test, and Dixon test; For different observation carriers, whether the observation carrier is abnormal is judged based on the change pattern of longitude and latitude over time. If the observation carrier is a ship, fixed-point or cruise observation is considered. If it is a surface drifting buoy, it should move with the ocean current. Fixed-point observation buoys normally only swing with the tide in a specific area. If abnormal movement is found, all data corresponding to the period will be judged as suspicious or abnormal. If the quality control inspection of longitude and latitude data finds abnormalities that cannot be corrected, the hydrological and meteorological observation data of the point will be marked as suspicious or erroneous.

5. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: In salinity correlation quality control, since seawater salinity is a function of parameters such as temperature, pressure, and conductivity, quality control is performed by performing one or more of the following tests on temperature and conductivity: blank value test, time test, location test, equipment log test, threshold test, triple standard deviation test, kurtosis test, Grubbs test, Dixon test, gradient test, rigid value test, and manual visualization test. Based on the quality control results of temperature and conductivity, salinity is subjected to correlation quality control. As long as one of the parameters of temperature, pressure, and conductivity is abnormal, the corresponding salinity is judged to be abnormal.

6. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: In the vector correlation quality control, the vector size is tested by one or more of the following tests: threshold test, triple standard deviation test, Grubbs test, Dixon test, gradient test, rigid value test and manual visualization test; the direction is tested by one or more of the following tests: threshold test and rigid value test. If one of the size and direction fails the test, the vector is judged to be suspicious or abnormal. The tested vector is decomposed into the east component and the north component, and then the two components are tested using one or more of the triple standard deviation test, Grubbs test, Dixon test, gradient test and stiff value test. If one of the two components fails the test, the vector is judged to be suspicious or abnormal.

7. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: In the quality control of relative humidity correlation, the atmospheric temperature is tested by one or more of the following tests: time test, location test, equipment log test, threshold test, triple standard deviation test, kurtosis test, Grubbs test, Dixon test, gradient test and stiff value test. If the test finds that the atmospheric temperature has an abnormal value, the corresponding relative humidity is also determined to be an abnormal value.

8. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: In the quality control of wind-wave-current correlation, the complex nonlinear interaction between wind, waves and currents is considered. When the wind changes greatly, sudden changes in waves and currents are allowed. According to the correspondence between wind speed and wave height specified in the national standard HY / T 0315-2021, combined with the curve distribution of wave height and maximum wind speed, it is judged whether the wave height data is abnormal to avoid misjudging the maximum wave height in special weather such as typhoons as abnormal values.

9. A method for quality control of hydrological and meteorological data using correlation as claimed in claim 1, characterized in that: Correlation quality control is also carried out on other factors that may have a large correlation, including temperature and dew point, weather phenomena and visibility, maximum wave height and effective wave height, maximum wave height and average wave height, flow velocity and ship's attitude data observed during navigation, flow velocity observed at anchor and the instrument's own attitude data, etc. According to the correlation characteristics between each factor, corresponding quality control inspection methods are used for quality control.

10. A method for quality control of hydrological and meteorological data using correlation according to any one of claims 1 to 9, characterized in that: During the quality control process, we should fully understand the observation sources and methods of each factor and determine whether there is any correlation between the factors to prevent excessive quality control of factors that have no correlation.

Images