Abnormal site identification method and system based on observation value and distance dynamic threshold

Abstract

The application discloses an abnormal station identification method and system based on observation values and distance dynamic threshold values, and relates to the technical field of observation station data quality control. The method comprises the following steps: acquiring verified qualified long sequence of hourly precipitation data of observation stations and observation station position data; constructing a relationship function of the distance of adjacent observation stations and the difference threshold value of hourly precipitation, and a relationship function of the hourly precipitation of observation stations and the difference threshold value of hourly precipitation of adjacent observation stations; on the basis, calculating the data anomaly probability of all observation stations in the analysis period; and judging the observation station grade according to the data anomaly probability. The application breaks through the limitation of traditional fixed threshold value by fusing the observation value size and the station distance to dynamically construct a double threshold function, can realize accurate and adaptive identification of abnormal data of observation stations, and improves the reliability and business application value of ground observation data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of observation station data quality control, and in particular to an abnormal station identification method and system based on observation value and distance dynamic threshold. BACKGROUND

[0002] China has built a nationwide, high-density ground meteorological and hydrological observation network, which provides key data support for rainstorm early warning, flood control and water resource management. However, due to long-term exposure to the outdoor environment, observation stations are easily affected by factors such as instrument aging, sensor blockage, lightning damage, communication interruption, strong wind disturbance, and snow cover, resulting in abnormal fluctuations, persistent high or low quality problems of observation data. The existing business generally uses a simple quality control method based on fixed threshold or extreme value judgment to determine that the observation results exceeding the threshold and the observation station are abnormal, or to set a unified tolerance range according to the historical standard deviation, which ignores the nonlinear influence of the observation object on the error tolerance. Taking precipitation observation as an example, the allowable error is usually not more than 2mm when it is raining, and the error of 5~10mm is still within a reasonable range. In addition, in actual meteorological observation, the hourly precipitation difference of adjacent observation stations is obviously spatially dependent due to factors such as local topography and precipitation system structure: the closer the observation stations, the more consistent they are affected by the same precipitation process, and the smaller the allowable hourly precipitation difference; on the contrary, the farther the observation stations, the higher the possibility of being affected by different precipitation units, and the larger the allowable hourly precipitation difference. However, the existing method treats all adjacent observation stations equally and uses a unified tolerance, resulting in a high false negative rate between close observation stations and a high false positive rate between distant observation stations. This processing method seriously restricts the accuracy and adaptability of abnormal identification.

[0003] Therefore, an intelligent identification method that can simultaneously fuse observation values and distances is needed to achieve precise and adaptive identification of abnormal observation data and observation stations, and to improve the reliability and business application value of ground observation data. SUMMARY

[0004] The purpose of the present application is to provide an abnormal station identification method and system based on observation value and distance dynamic threshold, which realizes precise and adaptive identification of abnormal observation data and observation stations, and improves the reliability and business application value of ground observation data.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides an abnormal station identification method based on observation value and distance dynamic threshold, comprising:

[0007] S1: Obtain hourly precipitation data and station location data for a long sequence of verified observation stations within the watershed;

[0008] S2: Based on the verified hourly precipitation data and station location data of the long sequence of observation stations, construct a relationship function between the distance to nearby observation stations and the hourly precipitation difference threshold. The relationship function between the hourly precipitation at the observation station and the threshold value of the difference between the hourly precipitation at neighboring observation stations. ;

[0009] S3: For any observation station in the period to be analyzed, record its hourly precipitation, its distance from all neighboring observation stations, and the difference in hourly precipitation.

[0010] S4: Substitute the hourly precipitation of the observation station mentioned in S3 and its distance from all neighboring observation stations into the relationship function. and relational functions Calculate the probability of data anomalies at the observation station to be analyzed;

[0011] S5: Determine the level of the observation station to be analyzed based on the probability of data anomalies, including observation stations of concern, suspected abnormal observation stations, or abnormal observation stations;

[0012] S6: Perform steps S3~S5 for all observation stations during the analysis period to obtain the location and observation data of all observation stations of interest, suspected, and abnormality.

[0013] Furthermore, in S2, for any verified observation station, observation stations within 50 km of it are designated as its neighboring observation stations.

[0014] Furthermore, in S2, the relational function The specific construction process is as follows:

[0015] Calculate the distances between all nearby observation stations And the hourly precipitation difference between two adjacent observation stations Forming the distance of nearby observation stations Difference from hourly precipitation point set ;

[0016] The maximum hourly precipitation difference was calculated within the distance intervals of 0-1km, 1-3km, 3-5km, 5-10km, 10-30km, and 30-50km between nearby observation stations. This maximum value was then multiplied by a coefficient of 1.1 and rounded up to obtain the hourly precipitation difference threshold for each distance interval. Based on this threshold, a relationship function between the distance between nearby observation stations and the hourly precipitation difference threshold was constructed. , It is a piecewise step function, in the form of:

[0017]

[0018] in, , , , , , These are the threshold values ​​for the hourly precipitation difference for the corresponding distance intervals.

[0019] Furthermore, in S2, the relational function The specific construction process is as follows:

[0020] Count the hourly precipitation at each observation station. and the difference between its hourly precipitation and that of nearby observation stations The hourly precipitation at the observation station was recorded. Difference from hourly precipitation point set ;

[0021] The hourly precipitation at the statistical observation stations was 0~10mm, 10~30mm, and 30~50mm, respectively. Within a 50mm interval, the maximum value of the hourly precipitation difference is multiplied by a coefficient of 1.1 and rounded up to obtain the hourly precipitation difference threshold for the corresponding hourly precipitation interval; a relationship function is then constructed between the hourly precipitation at one observation station and the hourly precipitation difference threshold between neighboring observation stations. , It is a piecewise step function, in the form of:

[0022]

[0023] in, , , , These are the threshold values ​​for the hourly precipitation difference within the corresponding hourly precipitation intervals.

[0024] Furthermore, in step S4, the method for calculating the probability of data anomalies at the observation station to be analyzed is as follows:

[0025]

[0026] In the formula, For the observation station to be analyzed The probability of data anomalies; For observation station The the number of neighboring observation stations of the observation station, the number of neighboring observation stations of the observation station the total number of neighboring observation stations of the observation station; the hourly precipitation of the observation station the hourly precipitation of the observation station the distance between the observation station and its neighboring observation stations the distance between the observation station and its neighboring observation stations the distance between the observation station and its neighboring observation stations the difference of hourly precipitation between the observation station and its neighboring observation stations the difference of hourly precipitation between the observation station and its neighboring observation stations the difference of hourly precipitation between the observation station and its neighboring observation stations The function represents a Boolean function, and the function value is 1 when the expression in the parentheses is true, otherwise 0.

[0027] Further, in the S5, when the observation station is marked as a normal observation station; when the observation station is marked as an observation station of interest; when the observation station is marked as a suspected abnormal observation station; when the observation station is marked as an abnormal observation station. the observation station is marked as an abnormal observation station. the observation station is marked as an abnormal observation station.

[0028] In a second aspect, the application provides an abnormal station identification system based on observation value and distance dynamic threshold, comprising:

[0029] A data acquisition module, which is used to acquire long sequence hourly precipitation data of observation stations and observation station position data in a verified basin range;

[0030] A function construction module, which is used to construct a relationship function of neighboring observation station distance and hourly precipitation difference threshold and a relationship function of observation station hourly precipitation and neighboring observation station hourly precipitation difference threshold according to the long sequence hourly precipitation data of observation stations and observation station position data acquired by the data acquisition module;

[0031] An abnormal probability calculation module, which is used to calculate the data abnormal probability of an observation station to be analyzed based on the relationship functions and constructed by the function construction module and the hourly precipitation data of the observation station to be analyzed and the distances between the observation station to be analyzed and all neighboring observation stations;

[0032] An observation station identification module, which is used to identify abnormal observation stations according to the data abnormal probability.​​

[0033] In a third aspect, the present application provides an electronic device, comprising:

[0034] a memory for storing executable instructions;

[0035] a processor for executing the executable instructions stored in the memory to implement the abnormal station identification method based on observation value and distance dynamic threshold.

[0036] In a fourth aspect, the present application provides a computer readable storage medium storing computer executable instructions, which are executed by a processor to implement the abnormal station identification method based on observation value and distance dynamic threshold.

[0037] In a fifth aspect, the present application provides a computer program product comprising a computer program or computer executable instructions, which are executed by a processor to implement the abnormal station identification method based on observation value and distance dynamic threshold.

[0038] The present application breaks through the limitation of traditional fixed threshold by fusing observation value size and station distance to dynamically construct a double threshold function, and realizes the adaptability of abnormal identification. Compared with the existing method, its advantages are: 1) accurate matching of precipitation intensity tolerance (strict control for light rain and relaxation for heavy rain); 2) introduction of spatial dependence, near distance station difference value threshold is more strict, and far distance is more relaxed, reducing the false alarm rate; 3) the segmented ladder function structure is simple and efficient, and is suitable for business system; 4) considering the data of multiple adjacent stations for probability calculation to prevent the influence of abnormal station on the judgment of adjacent normal stations; 5) quantifying abnormal probability and grading early warning, improving the intelligence and practicality of data quality control, and significantly enhancing the reliability of meteorological and hydrological data and the support ability of disaster prevention decision. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flowchart of an abnormal station identification method based on observation value and distance dynamic threshold provided by an embodiment of the present application.

[0040] Figure 2 A distance and hourly precipitation diagram of selected observation stations and adjacent observation stations provided by an embodiment of the present application.

[0041] Figure 3 A point set of distance and hourly precipitation difference of adjacent observation stations in an embodiment of the present application And a diagram of the relationship function. .

[0042] Figure 4 A distance and hourly precipitation diagram of a normal observation station and its adjacent observation stations provided by an embodiment of the present application.

[0043] Figure 5 This is a schematic diagram showing the distance and hourly precipitation of another normal observation station and its neighboring observation stations, provided for an embodiment of this application.

[0044] Figure 6 A schematic diagram showing the distance and hourly precipitation of an abnormal observation station and its neighboring observation stations provided in this application embodiment. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In one exemplary embodiment, such as Figure 1 As shown, an anomaly site identification method based on observation values ​​and a dynamic distance threshold is provided, including the following steps:

[0047] S1: Obtain hourly precipitation data and station location data for a long sequence of verified observation stations within the watershed.

[0048] S2: Based on the verified hourly precipitation data and station location data of the long sequence of observation stations, construct a relationship function between the distance to nearby observation stations and the hourly precipitation difference threshold. The relationship function between the hourly precipitation at the observation station and the threshold value of the difference between the hourly precipitation at neighboring observation stations. The specific construction process is as follows:

[0049] S2.1: For any verified observation station, observe stations within 50 km of it are designated as its neighboring observation stations, and the hourly precipitation of this observation station and all its neighboring observation stations is recorded. Figure 2 As shown;

[0050] S2.2: Calculate the distances between all nearby observation stations. And the hourly precipitation difference between two adjacent observation stations Forming the distance of nearby observation stations Difference from hourly precipitation point set ;

[0051] S2.3: Calculate the maximum hourly precipitation difference between nearby observation stations within the distance intervals of 0~1km, 1~3km, 3~5km, 5~10km, 10~30km, and 30~50km. Multiply this maximum value by a coefficient of 1.1 and round up to obtain the hourly precipitation difference threshold for the corresponding distance interval. Based on the hourly precipitation difference threshold for each distance interval, construct a relationship function between the distance between nearby observation stations and the hourly precipitation difference threshold. , For piecewise step functions, such as Figure 3 As shown, the form is:

[0052]

[0053] in, , , , , , These are the threshold values ​​for the hourly precipitation difference for the corresponding distance intervals;

[0054] S2.4: Calculate the hourly precipitation for each observation station. and the difference between its hourly precipitation and that of nearby observation stations The hourly precipitation at the observation station was recorded. Difference from hourly precipitation point set ;

[0055] S2.5: The hourly precipitation at the statistical observation stations were 0~10mm, 10~30mm, and 30~50mm respectively. Within a 50mm interval, the maximum value of the hourly precipitation difference is multiplied by a coefficient of 1.1 and rounded up to obtain the hourly precipitation difference threshold for the corresponding hourly precipitation interval; a relationship function is then constructed between the hourly precipitation at one observation station and the hourly precipitation difference threshold between neighboring observation stations. , It is a piecewise step function, in the form of:

[0056]

[0057] in, , , , These are the threshold values ​​for the hourly precipitation difference within the corresponding hourly precipitation intervals.

[0058] S3: For any observation station in the time period to be analyzed Record its hourly precipitation Its distance from all nearby observation stations and hourly precipitation difference .

[0059] S4: The observation station described in S3 hourly precipitation and its distance from all nearby observatories Substitute into the relational function respectively and relational functions Calculate the observation station to be analyzed Data anomaly probability :

[0060]

[0061] In the formula, For the observation station to be analyzed The probability of data anomalies; For observation station The A nearby observation station, For observation station The total number of nearby observation stations; For observation station Hourly precipitation; For observation station Its neighboring observation station The distance; For observation station Its neighboring observation station The hourly precipitation difference; The function represents a Boolean function. The function value is 1 when the expression inside the parentheses is true, and 0 otherwise.

[0062] S5: Determine the level of the observation station to be analyzed based on the anomaly probability of the data. At that time, the observation station Marked as a normal observation station; when At that time, the observation station Mark as an observation station of interest; when At that time, the observation station Marked as a suspected anomaly observation station; when At that time, the observation station Marked as an anomalous observation station.

[0063] S6: Perform steps S3~S5 for all observation stations during the analysis period to obtain the location and observation data of all observation stations of interest, suspected, and abnormality.

[0064] The following example, using an observation station in a province in Northeast my country, illustrates a method for identifying abnormal stations based on observation values ​​and a dynamic distance threshold, as provided in the above embodiments of this application. In this example, the method for identifying abnormal stations based on observation values ​​and a dynamic distance threshold specifically includes the following steps:

[0065] B1: Obtain hourly precipitation data and station location data for the past 5 years from over 7,000 verified and qualified observation stations in the province;

[0066] B2: Based on the verified hourly precipitation data and station location data of the long sequence of observation stations, construct a relationship function between the distance to nearby observation stations and the threshold of hourly precipitation difference. The relationship function between the hourly precipitation at the observation station and the threshold value of the difference between the hourly precipitation at neighboring observation stations. ;

[0067] B2.1: For any verified observation station, observe stations within 50 km of it shall be designated as neighboring observation stations, and the hourly precipitation of the observation station and all its neighboring observation stations shall be recorded.

[0068] B2.2: Calculate the distances between all nearby observation stations. And the hourly precipitation difference between two adjacent observation stations Forming the distance of nearby observation stations Difference from hourly precipitation point set ;

[0069] B2.3: Calculate the maximum hourly precipitation difference between nearby observation stations within the distance intervals of 0~1km, 1~3km, 3~5km, 5~10km, 10~30km, and 30~50km. Multiply this maximum value by a coefficient of 1.1 and round up to obtain the hourly precipitation difference threshold for the corresponding distance interval. Construct a function relating the distance between nearby observation stations to the hourly precipitation difference threshold. , The piecewise step function is expressed as:

[0070]

[0071] B2.4: Calculate the hourly precipitation for each observation station. and the difference between its hourly precipitation and that of nearby observation stations The hourly precipitation at the observation station was recorded. Difference from hourly precipitation point set ;

[0072] B2.5: The hourly precipitation at the statistical observation stations was 0~10mm, 10~30mm, and 30~50mm respectively. Within a 50mm interval, the maximum value of the hourly precipitation difference is multiplied by a coefficient of 1.1 and rounded up to obtain the hourly precipitation difference threshold for the corresponding hourly precipitation interval; a relationship function is then constructed between the hourly precipitation at one observation station and the hourly precipitation difference threshold between neighboring observation stations. , The piecewise step function is expressed as:

[0073]

[0074] B3: For a certain observation station at the current time ,like Figure 4 As shown, the recorded hourly precipitation was 35 mm. Within a 50 km radius, there were four nearby observation stations with precipitation amounts of 20 mm, 30 mm, 39 mm, and 36 mm, respectively; the corresponding hourly precipitation differences were 15 mm, 5 mm, 4 mm, and 1 mm. (Observation station...) The distances to these four nearby observation stations are 40km, 10km, 30km and 25km, respectively.

[0075] For a certain observation station at the current time ,like Figure 5 As shown, the recorded hourly precipitation was 35 mm. Within a 50 km radius, there were four nearby observation stations with precipitation amounts of 150 mm, 30 mm, 39 mm, and 36 mm, respectively; the corresponding hourly precipitation differences were 115 mm, 5 mm, 4 mm, and 1 mm. (Observation station...) The distances to these four nearby observation stations are 40km, 10km, 30km and 25km, respectively.

[0076] For a certain observation station at the current time ,like Figure 6 As shown, the hourly precipitation was recorded as 100 mm. Within a 50 km radius, there were four nearby observation stations with precipitation amounts of 20 mm, 30 mm, 39 mm, and 36 mm, respectively; the corresponding hourly precipitation differences were 80 mm, 70 mm, 61 mm, and 64 mm. The distances to these four nearby observation stations are 40km, 10km, 30km and 25km, respectively.

[0077] B4: Substitute the hourly precipitation of the observation station mentioned in S3 and its distance from all neighboring observation stations into the relationship function. and relational functions Calculate the probability of data anomalies at the observation station to be analyzed;

[0078] Observation station to be analyzed The probability of data anomalies is:

[0079]

[0080] Observation station to be analyzed The probability of data anomalies is:

[0081]

[0082] Observation station to be analyzed The probability of data anomalies is:

[0083]

[0084] B5: Determine the level of the observation station to be analyzed based on the probability of data anomalies.

[0085] Observation stations The probability of data anomalies is 0, therefore This is a normal observation station.

[0086] Observation stations The probability of data anomalies is 0.25, therefore This is also a normal observation station. The probability of its data being abnormal is not 0, which is due to the influence of the observation data of a nearby observation station at 150mm, possibly indicating that the data from that nearby observation station is abnormal.

[0087] Observation stations The probability of data anomalies is 1, therefore This is an anomaly observation station.

[0088] B6: Perform steps S3~S5 for all observation stations during the analysis period to obtain the location and observation data of all observation stations of interest, suspected, and abnormality.

[0089] Finally, it should be noted that the above embodiments are intended to illustrate the technical solutions of the present invention and do not constitute any limitation on the present invention. Those skilled in the art should fully understand that modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions for any part or all of the technical features are entirely feasible. Such modifications or substitutions, as long as they do not depart from the scope of protection defined by the claims of the present invention, should be considered reasonable extensions of the present invention.

Claims

1. An abnormal site identification method based on observation values ​​and dynamic distance thresholds, characterized in that, include: S1: Obtain hourly precipitation data and station location data for a long sequence of verified observation stations within the watershed; S2: Based on the verified hourly precipitation data and station location data of the long sequence of observation stations, construct a relationship function between the distance to nearby observation stations and the hourly precipitation difference threshold. The relationship function between the hourly precipitation at the observation station and the threshold value of the difference between the hourly precipitation at neighboring observation stations. Among them, relational functions The specific construction process is as follows: Calculate the distances between all nearby observation stations And the hourly precipitation difference between two adjacent observation stations Forming the distance of nearby observation stations Difference from hourly precipitation point set ; The maximum hourly precipitation difference was calculated within the distance intervals of 0-1km, 1-3km, 3-5km, 5-10km, 10-30km, and 30-50km between nearby observation stations. This maximum value was then multiplied by a coefficient of 1.1 and rounded up to obtain the hourly precipitation difference threshold for each distance interval. Based on this threshold, a relationship function between the distance between nearby observation stations and the hourly precipitation difference threshold was constructed. , It is a piecewise step function, in the form of: in, , , , , , These are the threshold values ​​for the hourly precipitation difference for the corresponding distance intervals; relational functions The specific construction process is as follows: Count the hourly precipitation at each observation station. and the difference between its hourly precipitation and that of nearby observation stations The hourly precipitation at the observation station was recorded. Difference from hourly precipitation point set ; The hourly precipitation at the statistical observation stations was 0~10mm, 10~30mm, and 30~50mm, respectively. Within a 50mm interval, the maximum value of the hourly precipitation difference is multiplied by a coefficient of 1.1 and rounded up to obtain the hourly precipitation difference threshold for the corresponding hourly precipitation interval; a relationship function is then constructed between the hourly precipitation at one observation station and the hourly precipitation difference threshold between neighboring observation stations. , It is a piecewise step function, in the form of: in, , , , These are the threshold values ​​for the hourly precipitation difference within the corresponding hourly precipitation intervals; S3: For any observation station in the period to be analyzed, record its hourly precipitation, its distance from all neighboring observation stations, and the difference in hourly precipitation. S4: Substitute the hourly precipitation of the observation station mentioned in S3 and its distance from all neighboring observation stations into the relationship function. and relational functions Calculate the probability of data anomalies at the observation station to be analyzed: In the formula, For the observation station to be analyzed The probability of data anomalies; For observation station The A nearby observation station, For observation station The total number of nearby observation stations; For observation station Hourly precipitation; For observation station Its neighboring observation station The distance; For observation station Its neighboring observation station The hourly precipitation difference; The function represents a Boolean function. The function value is 1 when the expression inside the parentheses is true, and 0 otherwise. S5: Determine the level of the observation station to be analyzed based on the probability of data anomalies, including observation stations of concern, suspected abnormal observation stations, or abnormal observation stations; S6: Perform steps S3~S5 for all observation stations during the analysis period to obtain the location and observation data of all observation stations of interest, suspected, and abnormality.

2. The abnormal site identification method based on observation values ​​and dynamic distance thresholds according to claim 1, characterized in that, In S2, for any verified observation station, observation stations within 50 km of it are designated as nearby observation stations.

3. The abnormal site identification method based on observation values ​​and dynamic distance thresholds according to claim 2, characterized in that, In S5, when At that time, the observation station Marked as a normal observation station; when At that time, the observation station Mark as an observation station of interest; when At that time, the observation station Marked as a suspected anomaly observation station; when At that time, the observation station Marked as an anomalous observation station.

4. An anomaly site identification system based on observation values ​​and a dynamic distance threshold, used to execute the anomaly site identification method based on observation values ​​and a dynamic distance threshold as described in any one of claims 1 to 3, characterized in that, include: The data acquisition module is used to acquire verified and qualified hourly precipitation data and station location data of long-sequence observation stations within the watershed. The function construction module constructs a relationship function between the distance to neighboring observation stations and the hourly precipitation difference threshold based on the empirically verified hourly precipitation data and observation station location data obtained by the data acquisition module. The relationship function between the hourly precipitation at the observation station and the threshold value of the difference between the hourly precipitation at neighboring observation stations. ; Anomaly probability calculation module, which is based on the relational function constructed by the function construction module. and relational functions The hourly precipitation data of the observation station to be analyzed and its distance from all nearby observation stations are used to calculate the probability of data anomalies at the observation station to be analyzed. The observation station identification module identifies abnormal observation stations based on the probability of data anomalies.

5. An electronic device, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the abnormal site identification method based on observation values ​​and dynamic distance thresholds as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the abnormal site identification method based on observation values ​​and dynamic distance thresholds as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the abnormal site identification method based on observation values ​​and dynamic distance thresholds as described in any one of claims 1 to 3 is implemented.

Images