Ground wave radar automatic processing method and system

Abstract

The invention discloses a ground wave radar automatic processing method and system, and the method comprises the steps: obtaining a receiving antenna pattern, calibrating the amplitude and phase of a channel according to the receiving antenna pattern, and obtaining the original data of ground wave radar observation data; performing data decoding on the original data to obtain first ground wave radar data; performing quality control on the first ground wave radar data to obtain second ground wave radar data; when the original data belongs to a plurality of ground wave radars, performing multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data; and assimilating a surface flow field of the ground wave radar through a data assimilation module according to at least one of the second ground wave radar data or the third ground wave radar data to obtain target data of the ground wave radar observation data. The method can enhance the accuracy of ground wave radar monitoring data, and can be widely applied to the technical field of ground wave radar data assimilation.

Description

Technical Field

[0001] The present invention relates to the technical field of ground wave radar data assimilation, and in particular to a ground wave radar automatic processing method and system. Background Art

[0002] The current radar automatic processing system has not yet covered the entire process of UAV calibration, data quality control, data fusion and data assimilation, lacking a systematic, closed-loop automated solution. Since the actual directional pattern is affected by the on-site environment and is not an ideal directional pattern, the accuracy of the measured data will be affected by the directional pattern during the inversion process of the software. For radar data quality control, the current data quality control method is committed to using a single method to identify occasional non-realistic radar velocity vectors. When the threshold set by a single algorithm is too strict, accurate data may be deleted by mistake. In addition, the current time interval for radar current assimilation is relatively large, and the impact of the assimilation time interval of several hours will soon disappear. The assimilation technology based on unstructured grids involves interpolation between unstructured grids and regular grids. Frequent interpolation will lead to interpolation errors, resulting in significant numerical error accumulation, which in turn leads to momentum non-conservation. As the model runs, this error will be further amplified and ultimately affect the calculation results of the model. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a ground wave radar automatic processing method and system, in order to solve at least one problem of the prior art. The present invention can enhance the accuracy of ground wave radar monitoring data.

[0004] To achieve the above-mentioned purpose, an embodiment of the present invention provides a ground wave radar automatic processing method in one aspect, the method comprising the following steps: obtaining a receiving antenna pattern, calibrating a channel amplitude and phase according to the receiving antenna pattern, and obtaining raw data of ground wave radar observation data; decoding the raw data to obtain first ground wave radar data; performing quality control on the first ground wave radar data to obtain second ground wave radar data; when the raw data belongs to multiple ground wave radars, performing multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data; and assimilating the surface flow field of the ground wave radar according to at least one of the second ground wave radar data or the third ground wave radar data through a data assimilation module to obtain target data of the ground wave radar observation data.

[0005] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a ground wave radar automatic processing system, comprising: a first module, used to obtain a receiving antenna pattern, calibrate a channel amplitude and phase according to the receiving antenna pattern, and obtain raw data of ground wave radar observation data; a second module, used to decode the raw data to obtain first ground wave radar data; a third module, used to perform quality control on the first ground wave radar data to obtain second ground wave radar data; a fourth module, used to perform multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data when the raw data belongs to multiple ground wave radars; and a fifth module, used to assimilate the surface flow field of the ground wave radar according to at least one of the second ground wave radar data or the third ground wave radar data through a data assimilation module to obtain target data of the ground wave radar observation data.

[0006] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method and system for automatic processing of ground wave radar, which obtains the original data of ground wave radar observation data by obtaining the receiving antenna pattern, calibrating the channel amplitude and phase according to the receiving antenna pattern; decoding the original data to obtain the first ground wave radar data; quality control the first ground wave radar data to balance the accurate identification of outliers and the false alarm rate to obtain the second ground wave radar data; when the original data belongs to multiple ground wave radars, multi-station data fusion is performed on the second ground wave radar data to obtain the third ground wave radar data; through the data assimilation module, the surface flow field of the ground wave radar is assimilated according to at least one of the second ground wave radar data or the third ground wave radar data, meeting the needs of rapid changes in ocean currents, avoiding frequent interpolation in the model in the model assimilation technology, and obtaining the target data of the ground wave radar observation data. The present invention can enhance the accuracy of ground wave radar monitoring data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments are introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 is a flow chart of a ground wave radar automatic processing method provided by an embodiment of the present invention;

[0009] Figure 2 is a schematic diagram of a multi-level real-time data quality control method provided by an embodiment of the present invention;

[0010] Figure 3 is a schematic diagram of standard deviation data distribution provided by an embodiment of the present invention;

[0011] Figure 4 is a schematic diagram of the box-and-line method provided by an embodiment of the present invention;

[0012] Figure 5 is a schematic diagram of an Isolation Forest provided by an embodiment of the present invention;

[0013] Figure 6 is a schematic diagram of a radar observation range provided by an embodiment of the present invention;

[0014] Figure 7 It is a schematic diagram of the ground wave radar assimilation technology route provided by an embodiment of the present invention;

[0015] Figure 8 is a schematic diagram of real-time search data provided by an embodiment of the present invention;

[0016] Fig. 9 is a schematic diagram of a Nudging assimilation process provided by an embodiment of the present invention;

[0017] Fig.10 This is a schematic diagram of a system Web display page provided by an embodiment of the present invention;

[0018] Fig.11 This is a schematic diagram of a data sequence on the left column of a system web page provided by an embodiment of the present invention;

[0019] Fig.12 is a schematic diagram of an example of displaying a data time series provided by an embodiment of the present invention;

[0020] Fig.13 It is a structural schematic diagram of a ground wave radar automatic processing system provided by an embodiment of the present invention;

[0021] Fig.14 It is a data coverage area diagram of two pairs of radar stations of Luchaogang-Jinshan and Jigujiao-Nanhui provided by an embodiment of the present invention;

[0022] Figure 15a-Figure 15b is a schematic diagram comparing the directional diagrams of the Jinshan station in two tests provided by an embodiment of the present invention;

[0023] Figure 16a-16b 1 is a schematic diagram comparing the chicken bone reef directional diagrams of two tests provided by an embodiment of the present invention;

[0024] Fig.17 It is a comparison chart of new data and radar after channel calibration at Luchaogang-Jinshan Station from November 8 to 11, 2023, provided by an embodiment of the present invention;

[0025] Fig.18 It is a comparison chart of the old data without channel calibration and radar at Luchaogang-Jinshan Station from November 8 to 11, 2023 provided by an embodiment of the present invention;

[0026] Fig.19is a schematic diagram of a flow field without radar assimilation provided by an embodiment of the present invention;

[0027] Fig. 20 is a flow field schematic diagram of radar assimilation + Nudging assimilation quality control data (std) provided in an embodiment of the present invention;

[0028] Fig.21 Schematic diagram of the flow field of radar assimilation + Nudging assimilation quality control data (box) provided by an embodiment of the present invention;

[0029] Fig. 22 It is a flow field schematic diagram of radar assimilation + Nudging assimilation quality control data (PyOD) provided by an embodiment of the present invention;

[0030] Fig.23 It is a flow field diagram of radar assimilation + Nudging assimilation quality control data (iforest) provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Before describing the embodiments of the present invention in detail, some nouns and terms involved in the embodiments of the present invention are first described, and the nouns and terms involved in the embodiments of the present invention are applicable to the following explanations.

[0032] High-frequency ground wave radar is a remote sensing device that uses high-frequency electromagnetic waves (usually between 3 and 30 MHz) to detect and monitor ocean dynamic processes such as ocean surface flow fields, wind fields, and wave heights.

[0033] Radar channel calibration ensures that the signal characteristics of all receiving and transmitting channels are consistent when the radar system collects ocean data, thereby improving the accuracy and reliability of the observation data.

[0034] The antenna pattern is a graph that describes the intensity distribution of electromagnetic waves radiated or received by the antenna in all directions in space.

[0035] QC, short for quality control, is an important process to ensure the accuracy, completeness and consistency of data.

[0036] Multi-station data fusion: Data from multiple ground wave radar sites can be fused to obtain more accurate and wider coverage ocean current field information.

[0037] Data assimilation is a technique that combines observational data with numerical models. Its purpose is to improve the accuracy of model predictions by fusing observational information.

[0038] Three-dimensional variational assimilation finds an optimal model state at a specific moment by minimizing the weighted error between the model background field (prior information) and the observed data.

[0039] Nudging assimilation is a relatively simple data assimilation method that gradually guides the state of the numerical model to approach the observed data, thereby improving the prediction effect of the model.

[0040] like Figure 1 As shown, an embodiment of the present invention provides a ground wave radar automatic processing method, which may include but is not limited to steps S100 to S500: S100, obtaining a receiving antenna pattern, calibrating a channel amplitude and phase according to the receiving antenna pattern, and obtaining raw data of ground wave radar observation data; S200, decoding the raw data to obtain first ground wave radar data; S300, performing quality control on the first ground wave radar data to obtain second ground wave radar data; S400, when the raw data belongs to multiple ground wave radars, performing multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data; S500, assimilating the surface flow field of the ground wave radar according to at least one of the second ground wave radar data or the third ground wave radar data through a data assimilation module to obtain target data of the ground wave radar observation data.

[0041] In step S100 of some embodiments, a directional pattern test method based on a drone can be adopted, that is, by designing a new portable signal source to replace a large transponder, the directional pattern measurement in any environment can be realized by combining the flexibility and maneuverability of the drone. Optionally, a high-precision signal source and a GPS recorder are carried on the drone platform, and the antenna directional pattern test is realized by moving around the antenna to be tested, and the drone directional pattern calibration is applied to the data processing flow of the high-frequency ground wave radar to realize unlimited measurement in various complex environments to ensure the accuracy of the radar data finally obtained. Then, the actual measured receiving antenna directional pattern is imported into the high-frequency ground wave radar inversion software, and the actual measured receiving antenna directional pattern is used as the initial condition during radar inversion, and the amplitude response characteristics of each antenna of the high-frequency ground wave radar to signals from different directions are considered, and the amplitude difference and phase difference of each antenna channel are obtained through the algorithm, and are substituted as the initial condition in the algorithm processing to offset the influence of channel inconsistency on orientation or beam synthesis, and complete the calibration of channel amplitude and phase, and further, carry out sea state information extraction, reduce the influence of antenna directional pattern distortion, and improve the accuracy of ocean current observation data.

[0042] In some embodiments, channel amplitude and phase calibration may include:

[0043] 1) Direction sequence: defines the azimuth sequence corresponding to the amplitude and phase sampling points within the coverage of the directional pattern measurement. The azimuth sequence ranges from 0° to 359° or -179° to 180°, and is usually sampled at intervals of 1°.

[0044] 2) Directional pattern information: Record the amplitude and phase information of each antenna channel relative to the reference antenna (the monopole antenna in the monopole cross-loop antenna and usually the first antenna in the array antenna). This information changes with the direction, so it needs to be sampled and recorded in different directions.

[0045] For example, the channel amplitude calibration process can be performed by measuring the amplitude and phase response of each antenna channel in different directions to obtain a series of data points; selecting a reference antenna, and comparing and calibrating the amplitude and phase responses of other antenna channels relative to this reference; calculating the amplitude and phase difference between each antenna channel and the reference antenna; using an algorithm to process the collected data, and calibrating each antenna channel according to the calculated amplitude and phase difference; further, inputting the calibrated results as initial conditions into the signal processing algorithm of the radar system. Through this calibration process, the detection performance consistency of the radar system in different directions can be ensured, and the error and interference caused by the inconsistency of the antenna channels can be reduced. The radar system can detect and track targets more accurately and provide higher quality observation data.

[0046] In step S200 of some embodiments, the first ground wave radar data is obtained by decoding the raw data. Exemplarily, the raw data of ground wave radar observation data of a single-station radial flow (CRC format) and a dual-station composite flow (SVC format) can be decoded. The scattered raw data can also be processed through a NETCDF format file to optimize the memory of a data storage device.

[0047] In step S300 of some embodiments, the first ground wave radar data is quality controlled to obtain the second ground wave radar data. Inspection methods such as signal-to-noise ratio inspection, range inspection, continuity inspection, and data integrity inspection can be used to construct a multi-level real-time data quality control method such as radar echo quality inspection, radial flow field quality inspection, and synthetic flow field quality inspection. Figure 2 As shown, Figure 2 An optional technology roadmap for the multi-layered real-time data quality control approach may include:

[0048] (1) Radar echo quality inspection: Radar echo quality inspection should be embedded in the radar data processing software, and the quality control parameters should be adjusted as needed. The echo quality inspection mainly uses the signal-to-noise ratio inspection method. If the signal-to-noise ratio of the monopole antenna channel of the portable high-frequency ground wave radar is lower than the threshold, or the signal-to-noise ratios of the two cross-loop channels are lower than the threshold at the same time, the data cannot pass the quality inspection.

[0049] (2) Synthetic flow field quality inspection: The synthetic flow field quality inspection method is similar to that of the radial flow field, mainly including extreme value inspection, spatial continuity inspection, and temporal continuity inspection.

[0050] (3) Radial flow field quality inspection: The radial flow field quality inspection methods mainly include format inspection, extreme value inspection, data valid position inspection, spatial continuity inspection, and time continuity inspection. Radial current extreme value inspection, spatial continuity inspection, and time continuity inspection require the accumulation of a certain amount of data, and only after statistical analysis can a reasonable threshold range be obtained. The time continuity variation range is obtained through the accumulated data, and then the future flow field data is predicted through harmonic analysis. According to the comparative test data, a reasonable threshold range is analyzed.

[0051] 1) Format check: Check whether the data file name is named according to the standard file name according to the data record file format specification; check whether the data record site code, coordinates, time, and working frequency are missing, and whether the values ​​are appropriate.

[0052] 2) Extreme value check: The maximum radial velocity threshold represents the maximum reasonable surface radial velocity of a given domain. When the radial velocity value is lower than the threshold, the data passes the check; otherwise, the data fails the check.

[0053] 3) Data valid position check: If the data coordinate position exists on an island or is on land or behind it, the data position is invalid and cannot pass the check.

[0054] 4) Spatial continuity check: The ocean current velocity is continuous within a certain spatial range. The ocean current velocity is compared with the median of the radial ocean current velocity within a given spatial range. If the difference is lower than the warning threshold, the data passes the check; if it is higher than the warning threshold but lower than the failure threshold, the data is suspicious; if the difference is higher than the failure threshold, the data fails the check.

[0055] 5) Time continuity check: The current velocity is continuous within a certain time range. Check the current velocity and compare it with the radial current velocity data (predicted value) at the same location within a certain time range. If the difference is lower than the warning threshold, the data passes the check; if it is higher than the warning threshold but lower than the failure threshold, the data is suspicious; if the difference is higher than the failure threshold, the data fails the check.

[0056] In order to ensure the best effect of data quality control, consider selecting appropriate temporal and spatial continuity quality control strategies based on terrain characteristics, data characteristics, and ocean anomalies. Comprehensively considered, as shown in Table 1, the following four different quality control methods can be selected to ensure the best data quality control effect in different scenarios.

[0057] Table 1

[0058]

[0059]

[0060] In some embodiments, step S300 may include but is not limited to steps S310 to S360: S310, obtaining the data type of the first ground wave radar data; S320, when the data type belongs to the first type, performing a first quality control process on the first ground wave radar data to obtain first quality control data; S330, when the data type belongs to the second type, performing a second quality control process on the first ground wave radar data to obtain second quality control data; S340, when the data type belongs to the third type, performing a third quality control process on the first ground wave radar data to obtain third quality control data; S350, when the data type belongs to the fourth type, performing a fourth quality control process on the first ground wave radar data to obtain fourth quality control data; S360, obtaining the second ground wave radar data according to the first quality control data, the second quality control data, the third quality control data and the fourth quality control data;

[0061] Among them, as shown in Table 1, the first quality control processing is standard deviation to identify outliers, the second quality control processing is box plot to identify outliers, the third quality control processing is multi-dimensional outlier detection, and the fourth quality control processing is machine learning to identify outliers.

[0062] In step S310 of some embodiments, the data type of the first ground wave radar data is obtained to determine which quality control method is applicable to the data. The first type is a situation where outliers do not appear frequently in the data, but the data integrity requirement is high; the second type is a situation where there are some natural fluctuations in the data, but it is still necessary to identify data points that significantly deviate from the normal range; the third type is a situation where outliers in the data appear as density differences; the fourth type is a situation where the number and feature dimensions of the data points are very high, and the relationship between the data points is not easy to be intuitively quantified by distance or density. According to different data types, different quality control methods can be matched in the subsequent steps. By selecting a suitable data quality control method to balance the accurate identification of outliers and the false alarm rate, and maximize the benefits of existing data quality control methods.

[0063] In step S320 of some embodiments, when the data type of the first ground wave radar data is the first type, that is, when outliers do not appear frequently in the data, but the data integrity requirement is high, the first ground wave radar data can be quality controlled according to the method of identifying outliers based on the standard deviation, thereby obtaining the first quality control data. Figure 3 shown.

[0064] In some embodiments, step S320 may include but is not limited to steps S3201 to S3204: S3201, obtaining the spatial sample capacity, the first observation data and the first average value of the first observation data based on the spatial data set of the first ground wave radar data; S3202, obtaining the first standard deviation of the first observation data based on the spatial sample capacity, the first observation data and the first average value; S3203, obtaining the first absolute value of the difference between the first observation data and the first average value; S3204, eliminating the first observation data corresponding to the first absolute value greater than twice the first standard deviation to obtain the first intermediate data.

[0065] In steps S3201 to S3204 of some embodiments, the required spatial data set is obtained, which may include but is not limited to wind speed data, temperature data, or any other type of spatial distribution data of a weather station. According to the spatial data set, the spatial sample capacity, the first observation data, and the first average value of the first observation data are obtained, and further, the first standard deviation of the first observation data is obtained. For each data point (i.e., the first observation data), the first absolute value of the difference between it and the first average value is calculated. It is determined whether the first absolute value is greater than twice the first standard deviation. If so, the data point is marked as an outlier. For outliers, common processing methods include deleting outliers, replacing outliers with averages or medians, or further investigating the causes of outliers. Exemplarily, outliers are identified by standard deviation, and data whose difference between the current velocity components u and v in the entire area and their mean at the same time is greater than 2 times the standard deviation are eliminated. The calculation process is as follows:

[0066]

[0067]

[0068] Among them, y i is the value of the i-th data point, i.e., the corresponding first observation data; is the average value of the ocean current velocity components u and v at all spatial points at the same time, i.e. the first average value; N 1 is the spatial sample capacity; δ 1 is the first standard deviation.

[0069] In some embodiments, step S320 may also include steps S3211 to S3214: S3211, obtaining the first time series sample capacity, the second observation data and the second average value of the second observation data based on the time series data set of the first ground wave radar data; S3212, obtaining the second standard deviation of the second observation data based on the first time series sample capacity, the second observation data and the second average value; S3213, obtaining the second absolute value of the difference between the second observation data and the second average value; S3214, eliminating the second observation data corresponding to the second absolute value greater than twice the second standard deviation to obtain second intermediate data.

[0070] In some embodiments, in steps S3211 to S3214, a required time series data set is obtained, such as wind speed or temperature measurement values ​​at different time points at a specific location. According to the time series data set, the first time series sample size, the second observation data, and the second average value of the second observation data (the average value on the time series) are obtained, and further, the second standard deviation of the second observation data is obtained. For each time point The second observation data is taken, and the second absolute value of the difference between the second absolute value and the second average value is calculated. It is determined whether the second absolute value is greater than twice the second standard deviation. If so, the second observation data at that time point is marked as an outlier. Exemplarily, outliers are identified by standard deviation, and outlier processing is performed on each data point (a certain position point in the data distribution space, i.e., a certain spatial point) in the time series, and data whose difference between the current velocity components u and v and their mean is greater than 2 times the standard deviation is eliminated. The calculation process is as follows:

[0071]

[0072]

[0073] in, for The second observation data at time; is the average value of the ocean current velocity components u and v at all time points for a certain spatial point, i.e., the second average value; N 2 is the sample size of the first time series; δ 2 is the second standard deviation.

[0074] In some embodiments, step S320 may also include but is not limited to steps S3221 to S3224: S3221, based on the time series data set of the first ground wave radar data, obtaining the second time series sample capacity, the third observation data and the third average value of the third observation data; S3222, based on the second time series sample capacity, the third observation data and the third average value, obtaining the third standard deviation of the third observation data; S3223, obtaining the third absolute value of the difference between the third observation data at the current moment and the third observation data at the adjacent moment; S3224, eliminating the third observation data corresponding to the third absolute value greater than twice the third standard deviation to obtain the third intermediate data.

[0075] In some embodiments, in steps S3221 to S3224, a required time series data set is obtained, such as wind speed or temperature measurement values ​​at different time points at a specific location. According to the time series data set, the second time series sample capacity, the third observation data, and the third average value (average value on the time series) of the third observation data are obtained, and further, the third standard deviation of the third observation data is obtained. Calculate the current time The third observation data and adjacent time The third absolute value of the difference between the third observation data and the third observation data. Determine whether the third absolute value is greater than twice the third standard deviation. If so, mark the third observation data at the current moment as an outlier. For outliers, common processing methods include deleting outliers, replacing outliers with mean values ​​or medians, or further investigating the causes of outliers, and then removing jump values. Exemplarily, outliers are identified by standard deviation, and data at a certain moment that differs from the previous and subsequent data by more than 2 standard deviations is eliminated. The purpose is to remove jump values. The calculation process is as follows:

[0076]

[0077]

[0078] Among them, y(t m2 ) is t m2 The third observation data at time; for The third observation data at time; is the average value of the current velocity components u and v at all time points for a certain spatial point, that is, the third average value; N 3 is the sample size of the second time series; δ 3 is the third standard deviation.

[0079] In step S330 of some embodiments, when the data type of the first ground wave radar data is the second type, that is, there are some natural fluctuations in the data, but it is still necessary to identify data points that significantly deviate from the normal range, the first ground wave radar data can be quality controlled by identifying outliers through a box plot, thereby obtaining second quality control data.

[0080] In some embodiments, step S330 may include but is not limited to steps S3301 to S3305: S3301, obtaining the lower quartile and the upper quartile of the first ground wave radar data; S3302, obtaining the difference between the upper quartile and the lower quartile to obtain the interquartile range; S3303, setting the lower edge according to the lower quartile and the interquartile range; S3304, setting the upper edge according to the upper quartile and the interquartile range; S3305, eliminating the first ground wave radar data that is smaller than the lower edge and the first ground wave radar data that is larger than the upper edge to obtain second quality control data.

[0081] In some embodiments, in steps S3301 to S3305, the first ground wave radar data is quality controlled by identifying outliers through a box plot, wherein a box plot is a statistical chart used to display the dispersion of a set of data, which can display the upper bound, lower bound, median, upper and lower quartiles, and outliers of a set of data. Figure 4 As shown, the box plot usually consists of five numerical points, namely the minimum value, the lower quartile (Q 1 ), median (Q 2 ), upper quartile (Q 3 ) and the maximum value. The data is divided into four parts by five numerical points: the position of the lower quarter, that is, the 25% value after the data is sorted from small to large is Q 1 ; The top quarter, that is, the 75% value after the data is sorted from small to large is Q 3 ; The middle position, that is, the 50% value after the data is sorted from small to large is Q 2 In addition, an outlier is defined as a value less than the lower edge (Q 1 -1.5IQR) or greater than the upper edge (Q 3 +1.5IQR), where IQR is the interquartile range IQR = Q 3 –Q 1 . By eliminating the outliers, the second quality control data can be obtained.

[0082] Outliers in radar data can be identified based on the box plot. The drawing of the box plot relies on actual data. There is no need to assume in advance that the data obeys a specific distribution form and there are no restrictive requirements on the data. It just truly and intuitively shows the original appearance of the data shape. On the other hand, the standard for judging outliers by the box plot is based on quartiles and interquartile ranges. Quartiles have a certain degree of resistance. Up to 25% of the data can become arbitrarily far away without greatly disturbing the quartiles. Therefore, outliers cannot affect this standard. The result of identifying outliers with box plots is relatively objective.

[0083] In step S340 of some embodiments, when the data type of the first ground wave radar data is the third type, that is, the outliers in the data appear as density differences, the first ground wave radar data can be quality controlled according to a multi-dimensional outlier detection method to obtain third quality control data.

[0084] In some embodiments, step S340 may include but is not limited to steps S3401 to S3406: S3401, performing feature extraction on the first ground wave radar data to obtain multi-dimensional features; S3402, performing data cleaning on the multi-dimensional features; S3403, setting an outlier ratio threshold; S3404, constructing an outlier model based on the multi-dimensional features after data cleaning; S3405, performing outlier detection on the multi-dimensional features based on the outlier ratio threshold and the outlier model to obtain a first detection result; S3406, based on the first detection result, eliminating outliers on the first ground wave radar data to obtain third quality control data.

[0085] In some embodiments, features that are helpful for anomaly detection are selected from the original data. These features may include but are not limited to time features (year, month, day, hour, etc.), spatial features (latitude and longitude), environmental features (ocean current, wind speed, waves), etc. The features may also be converted and scaled as necessary to meet the input requirements of the algorithm. The features are then cleaned, such as the processing of missing values. In addition, a reasonable outlier ratio threshold (fraction) needs to be determined. This value represents the proportion of expected outliers in the data set. This proportion can be used in subsequent outlier detection models to guide the algorithm to determine the boundaries of outliers. Then, the anomaly model of Pycaret is used to model the feature data set, create an outlier model, perform outlier detection on the features through the outlier model and the outlier ratio threshold, and mark the detected outliers to obtain the test results. In this test result, outliers are usually marked as 1 (or True), while non-outliers are marked as 0 (or False).

[0086] For example, the anomaly detection algorithm tool library (PyOD) can detect outliers from multiple feature dimensions of the data. The date feature of the time series data can be decomposed into multiple other time-related features, including but not limited to year, month, day, week, quarter and other time-related features, and then the time-related features are cleaned, such as the processing of missing values.

[0087] At the same time, it is also necessary to set an outlier ratio threshold (fraction) for use in the subsequent outlier detection model. Optionally, use Pycaret's anomaly model to model the extracted features, build an outlier detection model, and use the outlier model and the outlier ratio threshold to perform outlier detection on time-related features, and mark the detected outliers to obtain the test results. In this test result, outliers are usually marked as 1 (or True), while non-outliers are marked as 0 (or False). And based on the test result, outliers are eliminated to obtain the third quality control data.

[0088] In step S350 of some embodiments, when the data type of the first ground wave radar data is the fourth type, that is, the number of data points and the feature dimension are very high, and the relationship between the data points is not easy to be intuitively quantified by distance or density, the first ground wave radar data can be quality controlled by identifying outliers through machine learning, thereby obtaining fourth quality control data.

[0089] In some embodiments, step S350 may include but is not limited to steps S3501 to S3503: S3501, initialize the isolation forest model and train the isolation forest model; S3502, perform outlier detection on the first ground wave radar data according to the trained isolation forest model to obtain a second detection result; S3503, eliminate outliers on the first ground wave radar data according to the second detection result to obtain fourth quality control data.

[0090] In some embodiments, machine learning can use the intrinsic characteristics and distribution of data to automatically learn and determine which data points are abnormal without manually setting fixed thresholds or rules. At the same time, machine learning can also adapt to the changes and evolution of data, and capture new abnormal patterns by continuously updating the model. Machine learning models usually require a large amount of data for training to ensure their accuracy and generalization ability. High-frequency ground wave radar can provide rich data resources to meet the needs of machine learning training data. Optionally, Isolation Forest is a machine learning algorithm for outlier detection. Its basic idea is to use binary trees to isolate observation points. It assumes that outliers are a small number of data points that are significantly different from normal values, so outliers can be easily identified.

[0091] Exemplarily, the isolation forest model is initialized by setting the isolation forest algorithm parameters, such as the number of trees, maximum depth, etc. Then, the isolation forest model is trained using the ground wave radar data set, and the model will learn how to isolate data points based on feature values. The trained isolation forest model is then used to perform outlier detection on the first ground wave radar data set. The model will calculate a score for each data point to indicate the possibility that it is an outlier. Based on the score output by the model, the threshold of the outlier is determined. Further, the data point is marked as normal or abnormal and output as the second detection result. Based on the second detection result, the first ground wave radar data is eliminated for outliers to obtain fourth quality control data. Among them, if Figure 5 As shown in the figure, the logic of the isolation forest is as follows: a) Randomly select a feature and randomly select a split point between the maximum and minimum values ​​of the feature. b) Divide the data set into two subsets according to the split point. The left subset contains observation points less than the split point, and the right subset contains observation points greater than or equal to the split point. c) Repeat the above steps until each subset contains only one observation point or reaches the preset maximum depth. In this way, a binary tree is constructed, and each leaf node corresponds to an observation point. d) Calculate the path length from each observation point to the root node. This length reflects the number of divisions required for the observation point to be isolated. The shorter the path length, the easier it is for the observation point to be isolated and the more likely it is to be an outlier. e) In order to improve stability and accuracy, repeat the above process multiple times to generate multiple binary trees, and take the average path length as the final result. This forms an isolation forest model.

[0092] In step S360 of some embodiments, the first ground wave radar data is matched with different quality control methods for quality control processing, so as to obtain first quality control data, second quality control data, third quality control data and fourth quality control data, and then obtain second ground wave radar data. Optionally, for each quality control data, these quality control data can be merged. When merging data, it is necessary to ensure that all data sets are aligned in timestamps. For missing timestamps, you can choose to fill in missing values, or only keep common timestamps that exist in all data sets. If different quality control methods have different effects on the data, you can adjust parameters to balance the effects of each data set. For example, different weights can be set for different quality control methods, or parameters can be adjusted according to other indicators of data quality (such as signal-to-noise ratio, data stability, etc.).

[0093] In some embodiments, step S400 may include but is not limited to steps S410 to S440: S410, converting second ground wave radar data belonging to multiple ground wave radars into a common grid; S420, in the common grid, saving second ground wave radar data of non-overlapping areas in the observation areas of multiple ground wave radars as fourth observation data; S430, in the common grid, performing secondary vector synthesis on second ground wave radar data of overlapping areas in the observation areas to obtain fifth observation data; S440, merging the fourth observation data and the fifth observation data to obtain third ground wave radar data.

[0094] In some embodiments, for areas with multiple radars, since each radar has its own observation range and corresponding grid points, it is necessary to perform multi-station data fusion on the second ground wave radar data of the multiple radars. Exemplarily, the data of the multiple radars should be converted to the same set of public grids, and the second ground wave radar data should be converted to the corresponding public grid positions. The data in the public grid is identified and judged to determine the data status of each grid, including: whether there is data, data quality level, whether there is data overlap, etc. Then, the data under the public grid is fused, and for non-overlapping data, the result is saved as the fourth observation data; for overlapping data, such as Figure 6 As shown in the figure, these data are observed by multiple radar systems at the same time. For these areas, secondary vector synthesis is required. The weights are calculated according to the quality level of each radar data, and then these data are merged to obtain a comprehensive and more accurate vector flow result, namely the fifth observation data. The final vector flow result, namely the third ground wave radar data, is obtained by merging the fourth observation data and the fifth observation data. It includes both the direct observation results of the non-overlapping areas and the weighted synthesis results of the overlapping areas, thereby ensuring that all available data are utilized and, where possible, improving the accuracy and reliability of the data through synthesis. The final vector flow result is a comprehensive data set that integrates all radar data, which reflects the flow field conditions of the entire observation area.

[0095] Optionally, the multi-station data fusion algorithm may include but is not limited to weighted average method, Kalman filter method, wavelet transform method, etc., which are not limited here. Then the multi-station data fusion algorithm design:

[0096] 1) Weighted average method

[0097] According to the data quality of each radar station, the velocity components u and v are weighted averaged respectively. The weighted average algorithm is as follows:

[0098]

[0099] Where U 1 ,U 2They are the data to be fused, namely the velocity components u and v; w1 and w2 are U 1 and U 2 mergeU is the data quality weight of the fusion result.

[0100] 2) Kalman filter method

[0101] According to the data quality of each radar station, the velocity components u and v are fused using Kalman filtering. The Kalman filter will calculate the optimal weight based on the uncertainty information of the provided data, and then perform a weighted average of the two signals to obtain a signal that is closer to the actual signal. This means that not only the radar observation results are needed, but also the uncertainty corresponding to the data (i.e., measurement error, where the error can be expressed by the radar signal strength).

[0102] Kalman filtering is divided into two steps: prediction and update.

[0103] predict:

[0104]

[0105]

[0106] renew:

[0107]

[0108]

[0109]

[0110] In the formula, and The posterior state estimates at time k-1 and time k are one of the filtering results, that is, the updated result, also called the optimal estimate; represents the prior state estimate at time k, that is, predicting the state at time k based on the optimal estimate at time k-1; P k-1 and P k are the posterior estimated covariances at time k-1 and time k, respectively. and The covariance of represents the state uncertainty and is one of the filtering results; is the prior estimated covariance at time k ( covariance of K k is the Kalman gain; A k is the state transfer matrix; B k is the control variable matrix; u k is the state control vector; Q is the system noise covariance matrix; z kis the measurement vector; H k is the transformation matrix from state vector to measurement vector; (·) T Represents matrix transpose.

[0111] 3) Wavelet transform method

[0112] Wavelet transform is a signal processing and data analysis technique used to analyze time domain (time domain) and frequency domain (frequency domain) characteristics. It can decompose the signal into components of different scales (frequencies), which makes it widely used in many fields, including image processing, audio processing, data compression, signal denoising, feature extraction, etc.

[0113] In step S500 of some embodiments, an unstructured grid three-dimensional variational assimilation module is developed based on FVCOM, a three-dimensional variational scheme is used in space to assimilate the surface current, and a Nudging scheme is used in time to extend the assimilation effect of the three-dimensional variational to the time dimension. Through the assimilation module, the surface flow field of the ground wave radar is assimilated according to at least one of the second ground wave radar data or the third ground wave radar data, so as to obtain the target data of the ground wave radar observation data.

[0114] In some optional embodiments, a universal radar assimilation system can be constructed based on this assimilation module, which can automatically obtain radar data on the server and complete data processing and quality control, and then perform surface flow field assimilation. Different assimilation parameters are set according to different types of radars, which has strong scalability.

[0115] Exemplarily, the assimilation module includes the following methods:

[0116] 1) Three-dimensional variational assimilation method

[0117] The three-dimensional variational (3DVAR) assimilation method obtains an analytical solution by minimizing the objective functional:

[0118]

[0119] In the formula, the objective function J(X) includes the model space J b and observation space J o Two terms, x in the model space term is the optimal solution of the objective function, that is, the assimilated model analysis field x a ; Background field x b is an N-dimensional vector; B is the N×N background error covariance matrix; (·) -1 represents matrix inversion; (·) T represents matrix transpose; in the observation space, H is the operator that maps the model space to the observation space, y 0 represents the observed value, and R is the observation error covariance matrix.

[0120] In the actual solution process, the method of solving the gradient is usually used to solve the minimum value of the objective function.

[0121]

[0122] By selecting a suitable optimization algorithm, the optimal solution of the objective function is obtained through iteration. Directly solving x has a large computational cost. If the incremental method is used, let δx = xx b The calculation cost can be greatly reduced, so formula (1) can be transformed into the following formula:

[0123]

[0124] d=H(x b )-y 0 (4)

[0125] The gradient expression of the objective function is transformed into the following form:

[0126] Where H T It is a tangent operator or adjoint operator. When solving the gradient function, an iterative method is usually used to solve the objective function and its gradient to obtain the final analytical increment result.

[0127] Since the B matrix is ​​a very large and almost ill-conditioned matrix (more than 10 6 ×10 6 ), it is difficult to solve the inverse of the matrix in practice. Usually, a series of simplifications are performed to reduce the computational cost. The B matrix can be expressed as: B = <x′x′ T >=<(x t -x b )(x t -x b ) T > (6)

[0128] Among them, x t is the true value. Since the true value is unknown, the difference between the two forecast fields is generally used to replace x. t -x b ,Right now

[0129] x′=x t1 -x t2 (7)

[0130] In the above formula, x t1 With x t2 Forecast the same time t at different times 1 With t 2 For the regional model, the forecast field at time t1 and t 2 Generally, it is 12 hours and 24 hours. As for the observation error covariance matrix R, it is usually assumed that the observation points are independent of each other and satisfy the Gaussian distribution. Therefore, R can be expressed as a diagonal matrix, and the main diagonal of the matrix is ​​the error variance σ 2 ,Right now:

[0131]

[0132] 2) Nudging

[0133] Nudging is a continuous assimilation method that can act on every step of the model calculation, and the process will not cause serious discontinuity problems. The embodiment of the present invention only considers the assimilation of the surface flow field. In order to prevent the generation of large velocity shear before and after assimilation (especially the velocity shear generated before assimilation and at the assimilation time), the Nudging assimilation scheme in the FVCOM model is introduced during the assimilation interval to eliminate the generation of velocity shear:

[0134]

[0135] α=α(x,y,z,t) (10)

[0136] F=F(α,x,y,z,t)(11)

[0137] W i =W i (x,y,z,t)(12)

[0138] In the formula, α o is the observed value; is the model prediction value; M is the number of observation points in the region; γ i is the weight factor of the ith observation point (0-1); G α Nudging factor, which aims to ensure a slow physical adjustment process. W i (x, y, z, t) can be specifically decomposed into horizontal, vertical, time and direction weights, respectively represented by w xy 、w σ 、w t 、w θ The relationship between them satisfies: W i (x,y,z,t)=w xy ·w σ ·w t ·w θ (13)

[0139] These four weight parameters can be expressed as:

[0140]

[0141]

[0142]

[0143]

[0144] In the formula, R 1 is the search radius; is the distance between the observation position and the model grid; R σ is the vertical search radius; T w is half of the assimilation time window, t is the current running time of the model, and t o is the time of three-dimensional variational assimilation; Δθ is the direction difference between the local isobath and the calculation point; c 1 is a constant ranging from 0.05 to 0.5.

[0145] Since only time is considered and the three-dimensional variational assimilation in the embodiment of the present invention acts on the model grid, the scheme can ignore the three weights of horizontal, vertical and direction, so: W i (x,y,z,t)=W i (t) = w t (18)

[0146] In the embodiment of the present invention, since the assimilation result at time t is considered to be applied to the times before and after assimilation, the α in the original equation o is replaced by the assimilation result at time t. Finally, it can be expressed as:

[0147] The numerical model used in the embodiment of the present invention is an ocean numerical model of unstructured grid, finite volume, free surface, and three-dimensional primitive equations, namely Finite-Volume Community Ocean Model (FVCOM). The unstructured grid used by FVCOM can better fit the coastline, and the resolution of different areas can be flexibly adjusted according to the research focus, saving computing resources and improving computing efficiency. The control equations of FVCOM are as follows:

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155] ρ=ρ(T,S,P) (27)

[0156] Where (x, y, z) are the easting, northing and vertical coordinates respectively; t is time; (u, v, w) are the velocity components in the (x, y, z) directions respectively; T and S are temperature and salinity respectively; f is the Coriolis parameter; ρ o is the water flow density; K m , K h are the vertical eddy viscosity coefficient and the thermal vertical eddy diffusion coefficient respectively; P is the total pressure, which is calculated by P h , P a , q, which are hydrostatic pressure, air pressure and non-hydrostatic pressure respectively; F u 、F v 、F w are momentum diffusion terms in the (x, y, z) directions; F T 、F S are temperature and salinity diffusion terms, respectively.

[0157] In some embodiments, a three-dimensional variational assimilation module based on FVCOM is established, and the process of the entire module can be roughly divided into the following steps: parameter setting, model operation, assimilation, update, and next cycle. Considering that there are data of different radar models, different radar parameters need to be provided for the model, including the name of the radar station, the observation grid range of the radar, the spatial resolution, and the interpolation search radius (as shown in Table 2); after the model calculation is completed, the flow field calculated by the model is extracted and entered into the assimilation module, and at the same time, it is determined whether there is radar data at this moment. As long as there is data from one or more radar stations, the flow field data will be assimilated, otherwise the model calculation at the next moment will be entered; after the assimilation is completed, the flow field of the model is updated and the next calculation of the model is entered. As the model runs, this process will continue to cycle.

[0158] Table 2

[0159] parameter meaning Radar_name Radar Station Min_lon Radar observation left boundary Max_lon Radar observation right boundary Min_lat Radar observation lower boundary Max_lat Radar observation upper boundary dlon Longitude resolution dlat Latitude resolution RD Influence Radius

[0160] Based on the established assimilation module, a radar assimilation scheme is established, such as Figure 7As shown in the figure, the scheme mainly includes three parts: the first part is the automatic download and preprocessing of radar data, which can automatically download the data of the day from the high-frequency ground wave radar server every day, and perform quality control, and finally obtain the assimilation input data required by the model; the second part is the assimilation part of the model, which can be divided into two stages. In the model initialization stage, the interpolation coefficient from the observation space to the model space is calculated through the radar observation network. In the model operation stage, it searches and determines whether there is radar data in real time, and assimilates the radar data when there is radar data; the third part is the Nudging assimilation part, which is responsible for expanding the radar assimilation results in space to the time dimension to solve the shear problem of flow velocity in time.

[0161] For example, in the first stage of assimilation, preprocessing is performed before assimilation. The radar data is obtained, and according to the given number of radars and interpolation radius, the radar data is processed and interpolated into the model space to obtain the observation field. A simple inverse distance weight is used for calculation:

[0162] In the formula, O i represents the i-th radar grid point found in the search radius, w i is the weight coefficient, which is determined by the inverse of the distance between the radar position and the model grid point, and n is the number of observation points found in the interpolation radius.

[0163] For FVCOM model grids, real-time search data can be used Figure 8 To express, Figure 8 The black triangle in the figure is the model mesh. In the FVCOM model, variables u and v are stored at the center of the triangle ( Figure 8 The red dots in the figure are used as the reference points, while some scalars such as temperature and salinity are stored at the vertices of the triangular mesh. The gray dashed line is the radar observation grid. The radar observation has the same resolution in the longitude and latitude directions, so each unit of the grid is a square, and the radar data is stored at the vertices of these squares. The red dashed circle is the search range, and its radius is RD. For each model grid, a search is performed for nearby radar data. For example, for the model triangle center points M1 and M2, there are 4 points falling within their search ranges ( Figure 8 According to the radar data searched, y is calculated according to formula (28) 0 .

[0164] The second stage of the assimilation part is the three-dimensional variational assimilation in space, including the construction of the target functional and its gradient, spatial smoothing and minimization iteration. The spatial smoothing operator is used to solve the truncation error caused by interpolation, and the minimization iteration uses the Fortran subroutine of L-BFGS-B. L-BFGS-B is a limited memory algorithm for solving simple boundary variables of large nonlinear optimization problems. It is suitable for problems such as the Hessian matrix (the second-order derivative of the target functional) is too large, unstructured, or too dense to be solved, and can quickly calculate so that the target function converges.

[0165] In order to cope with the needs of business operations and the subsequent access to other radar data, the number of radars that need to be assimilated can be set according to the actual number of radars. This solution can cope with the differences in data formats caused by different radar models (such as different observation space resolutions of different radar models). At the same time, for each input radar, there is a search radius parameter, which is the maximum range radius for interpolating radar data to the model grid to cope with the changes in the accuracy of nearshore and offshore grids in different radar area models. It should be noted that when the model is assimilated, the assimilation of each radar is independent of each other, that is, the lack of data from a certain radar station will not affect the assimilation of other radars. Applying this solution to the offshore area model can further optimize the model forecast results by assimilating radar data, and can continuously access new radar data. With the continuous access of coastal radar data, the accuracy of the model can be greatly improved.

[0166] The third stage of the assimilation is Nudging. Assuming the radar time resolution is 20 minutes, the Nudging section can refer to Fig. 9 To express the process, Fig. 9 The small and medium red dots are the moments when the flow field data is stored during the model operation, the large red dots are the radar assimilation moments, the blue dots are each time step of the model calculation, and 2Tw is the Nudging assimilation time window, which is set to 60s. First, the model calculation and the three-dimensional variational assimilation part of the radar (space) are saved according to the set time window (the time window 2Tw is set to 60s) during the model operation (the current results of the model operation in formula (19) ),exist Fig. 9 The moment is indicated by a small red dot in the figure, and the result after three-dimensional variational assimilation (α in formula (19)) is saved. o ),exist Fig. 9The big red dot indicates the moment. Assuming that the radar assimilation time interval is 20 minutes, the data before and after the radar assimilation is used as a cycle for Nudging assimilation (time). After the three-dimensional variational assimilation in space is completed, return to the front of the time window and perform nudging assimilation in time at each time step. According to the set time window, at each moment of saving the ocean current results ( Fig. 9 Red dot) for front and rear T w The data within the time is corrected to achieve the effect of transition in time. Fig. 9 The part selected by the red rectangle in the figure, and the model result saved by the red dot in the figure The difference between the assimilation result and the model result is The model results are corrected at each time step.

[0167] In some embodiments, in order to avoid errors caused by interpolation, when considering the observation term, the observed data is mapped to the model grid, and the original observation space is converted to the model space, so that the entire assimilation is performed in the model space. The radar observation results are interpolated onto the model grid, and Hδx+d in the cost function observation term is rewritten as δx+d 0 , where d 0 =x b -H(y 0 ), the observation term of the cost function is:

[0168] The cost function is expressed as:

[0169] In the high-frequency ground wave radar assimilation process, considering only the assimilation of the flow field, substituting the velocities u and v, the cost function will be expressed as a structure similar to the above formula:

[0170]

[0171]

[0172] Where δu and δv are the increments of the two directional components of the background velocity, u b and v b Represents the two directional components of the background field velocity, y u0 and v0 is the velocity component observed by the radar.

[0173] The background error covariance will be decomposed into: B = ∑C∑ (33)

[0174] In the formula, Σ is the root mean square error matrix of the model, which is a diagonal matrix whose elements are all root mean square errors; C is the autocorrelation matrix. This decomposition can give the root mean square error and correlation matrix respectively. Then:

[0175] Where, L x With L y are the relevant scales in the longitude and latitude directions, respectively; Δx and Δy are the distances between two locations in the longitude and latitude directions of the model, respectively.

[0176] The embodiment of the present invention only assimilates the surface flow field of the radar, and does not involve other variables. Therefore, there is no need to consider the covariance relationship between different variables in the B matrix, so a simplified B matrix is ​​used, which is a symmetric matrix, and the elements on the diagonal are the variances of the model errors. The observation error covariance matrix is ​​set to a diagonal matrix with the observation error variances on the diagonal.

[0177] In the embodiment of the present invention, a ground wave radar data visualization platform is also designed, such as Fig.10 The system web page shown is as follows: Fig.11 The data sequence on the left column of the page shown is as follows Fig.12 The data time series display example shown in Figure 3 is a visualization platform that can read and convert the mine radar data to general types to adapt to all mainstream WMS systems, and design and develop the front-end and back-end of the display system to provide real-time rendering efficiency. The ground wave radar data visualization platform is a comprehensive ground wave radar data processing, analysis, and visualization tool that aims to provide users with an efficient, accurate, and real-time data quality control, quality assessment, and multi-station data fusion solution. The system architecture of the visualization platform is shown in Table 3.

[0178] Table 3

[0179]

[0180]

[0181] Exemplarily, the data visualization content in the visualization platform may include but is not limited to:

[0182] 1) Original data display:

[0183] Single-station radial flow: Displays radar data for a single location, including speed, direction, and intensity. The length of the arrow indicates the intensity and speed of the data.

[0184] Dual-station composite flow: Displays radial flow data from two stations, including speed, direction, and intensity. The length of the arrow indicates the intensity and speed of the data.

[0185] Time series plot: Displays the temporal variation trend of raw data for selected spatial locations within the coverage of each single-station radial flow and dual-station composite flow.

[0186] 2) Comparison and display of original data and data after various quality controls:

[0187] Dual-station spatial plane flow vector diagram: The original data of the two stations and the quality-controlled data are combined and displayed on the same plane. The color is also distinguished by double-layer overlay.

[0188] Time series graph: Displays the temporal variation of the original data and the quality-controlled data for the selected spatial location.

[0189] 3) Multi-station data fusion display:

[0190] Space plane flow vector diagram: Combines the quality-controlled data from multiple sites to display the fused space plane flow vector diagram, showing the flow speed and direction in a larger range.

[0191] Time series graph: Displays the temporal variation trend of the fused data for the selected spatial location.

[0192] like Fig.13 As shown, an embodiment of the present invention further provides a ground wave radar automatic processing system, which can implement the above-mentioned ground wave radar automatic processing method. The system integrates a radar data processing and visualization platform and a data assimilation model, and includes: a first module, which is used to obtain a receiving antenna pattern, calibrate the channel amplitude and phase according to the receiving antenna pattern, and obtain the original data of the ground wave radar observation data; a second module, which is used to decode the original data to obtain the first ground wave radar data; a third module, which is used to perform quality control on the first ground wave radar data to obtain the second ground wave radar data; a fourth module, which is used to perform multi-station data fusion on the second ground wave radar data to obtain the third ground wave radar data when the original data belongs to multiple ground wave radars; and a fifth module, which is used to assimilate the surface flow field of the ground wave radar according to at least one of the second ground wave radar data or the third ground wave radar data through the data assimilation module to obtain the target data of the ground wave radar observation data.

[0193] The ground wave radar automatic processing system was successfully applied to the four radar base stations deployed by the Shanghai Marine Environment Monitoring and Forecasting Center in Hangzhou Bay in 2023, namely Nanhuizui, Jigujiao, Jinshan and Luchaogang. The radar model is OSMAR-S portable high-frequency ground wave radar. Fig.14 The following is a map of the data coverage area of ​​two pairs of radar stations: Luchaogang-Jinshan and Jigujiao-Nanhui. The results of the radar channel calibration scheme are as follows: Fig.15a Drone tests and results in July 2022, as shown, Fig.15b As shown in the drone test and results in October 2023, the overall distortion of the directional diagram of Jinshan Station is small and close to the ideal directional diagram. Fig.16a The directional pattern test and results in July 2022 are shown. Fig.16b The directional pattern test and results in October 2023 are shown. The directional pattern of Chicken Bone Reef is distorted, which is quite different from the ideal directional pattern. There is obvious prominent distortion. The results of the two tests of Chicken Bone Reef are basically the same, indicating that the electromagnetic environment is relatively stable, but the interference is large, resulting in serious distortion of the pattern. Among them, Figure 15a-Figure 15b , Figure 16a-16b The dotted line in the figure is the ideal directivity diagram, and the solid line is the measured directivity diagram.

[0194] According to the test results, the radar signal processing software (OSMAR-S Remotes iteso le) was substituted, the flow chart software of each single station was upgraded, and the changes in the data results after the software upgrade were compared. Fig.17 , Fig.18 The comparison chart shown in Figure 4 shows the comparison results between the Luchaogang-Jinshan station data and the radar from November 8 to 11, 2023.

[0195] Table 4

[0196] Flow direction (new) Flow rate (new) Flow direction (old) Flow rate (old) Mean square error 70.28 40.14 92.52 39.06 Relevance 0.75 0.22 0.54 0.13

[0197] Using the established radar assimilation module, a 15-day assimilation experiment was conducted on the data processed by the visualization platform, including 5 cases. The different radar parameters are shown in Table 5, and the conditions set for experimental Case 1-Case 5 are shown in Table 6.

[0198] Table 5

[0199] parameter meaning Radar_name JG Min_lon 121.20°E Max_lon 122.38°E Min_lat 30.30°N Max_lat 31.20°N dlon 0.03° dlat 0.03° RD 10km

[0200] Table 6

[0201] experiment set up Case 1 No radar assimilation Case 2 Radar assimilation + Nudging assimilation quality control data (std) Case 3 Radar assimilation + Nudging assimilation quality control data (box) Case 4 Radar assimilation + Nudging assimilation quality control data (PyOD) Case 5 Radar assimilation + Nudging assimilation quality control data (iforest)

[0202] like Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23The flow field diagrams at typical moments of the assimilation model based on five cases are shown. It can be obtained that, compared with Case 1, radar assimilation affects the flow field structure of the model in Case 2-5. Compared with Case 1 without radar assimilation, other cases show more complex flow field characteristics and are closer to the real situation. Case 2 to Case 5 all consider radar assimilation and Nudging assimilation, but use different quality control methods (STD, box, PyOD and iforest). The data ranges generated by different quality control processing are different in time and space, which will lead to local differences in the flow field. For example, the flow velocity in the area marked by the green circle in the flow field diagram increases or decreases, or the direction of the streamline changes, but the overall trend of the flow field is similar. Among them, STD quality control (Case 2) and Box quality control (Case 3) are relatively basic and common quality control methods, which can effectively filter out most of the outliers. The flow field may be more accurate overall after assimilation. However, it may appear broken or incoherent in some areas. PyOD quality control (Case 4) and I forest quality control (Case 5) are both suitable for more complex patterns, and quality control has better details in local areas. However, some real phenomena may be mistakenly deleted, especially in complex terrain areas. When using different quality control methods, the details and characteristics of the flow field may change, but the overall pattern remains the same. In order to obtain the most accurate model prediction, it is key to choose the appropriate quality control method based on terrain, data attributes, abnormal phenomena, etc.

[0203] In addition, the time series data is obtained at a specific location. The red dotted line in the figure represents the observation data of the ground wave radar, while the blue solid line represents the data predicted by the model. Fig.19 It can be clearly seen that there is a large difference between the flow field without data assimilation and the radar data. This is mainly reflected in the amplitude and periodicity of the two: there is a lack of consistency between them, indicating that there is a deviation between the model prediction and the actual observation without data assimilation. From the results after the model assimilated the data, it is obvious that in most time periods, the amplitude and periodicity of the model data and the ground wave radar measured data show similar trends, which shows that the assimilation process significantly improves the prediction accuracy of the model.

[0204] An embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned ground wave radar automatic processing method when executing the computer program. An embodiment of the present invention further provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned method when executed by the processor. An embodiment of the present invention further provides a computer program product or a computer program, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned method.

[0205] In summary, the ground wave radar automatic processing method and system according to the embodiment of the present invention have the following advantages:

[0206] 1. The present invention adopts a flexible pattern test method based on drones, which can be used for measurement in any environment, has high economic benefits and is easy to operate. Compared with uncalibrated radar, the data accuracy is significantly improved. The test shows that the correlation between flow velocity and flow direction is increased by about 0.1 and 0.2 respectively, and the root mean square error is reduced by 0.01m / s and 22.2°.

[0207] 2. The present invention proposes a flexible data quality control and fusion algorithm, which can be selected according to the terrain, data characteristics and marine anomalies, adapting to the optimal processing in different scenarios, allowing the best algorithm to be selected according to specific monitoring conditions, and optimizing data processing accuracy and efficiency. The present invention also develops a visualization platform based on the WMS architecture, establishes an automated process of "data acquisition-quality control-data fusion-automatic update-real-time display", and enhances the flexibility and real-time performance of data processing.

[0208] 3. The three-dimensional variational assimilation module constructed based on FVCOM can automatically acquire radar data and perform quality control, fusion and assimilation. Different assimilation parameters can be set for different types of radars. The system is highly scalable and meets business needs.

[0209] 4. The present invention integrates UAV calibration, data quality control, multi-station fusion and data assimilation into an automated system, realizing full process automation from data acquisition to assimilation, greatly improving the efficiency and accuracy of the radar system.

[0210] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A ground wave radar automatic processing method, characterized in that: The following steps are involved: Obtaining a receiving antenna pattern, calibrating channel amplitude and phase according to the receiving antenna pattern, and obtaining raw data of ground wave radar observation data; Decoding the original data to obtain first ground wave radar data; performing quality control on the first ground wave radar data to obtain second ground wave radar data; When the original data belongs to multiple ground wave radars, performing multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data; The surface flow field of the ground wave radar is assimilated by the data assimilation module according to at least one of the second ground wave radar data or the third ground wave radar data to obtain target data of the ground wave radar observation data.

2. The automatic processing method of ground wave radar according to claim 1, characterized in that: The step of performing quality control on the first ground wave radar data to obtain second ground wave radar data comprises the following steps: Acquire the data type of the first ground wave radar data; When the data type belongs to the first type, performing first quality control processing on the first ground wave radar data to obtain first quality control data; When the data type belongs to the second type, performing a second quality control process on the first ground wave radar data to obtain second quality control data; When the data type belongs to the third type, performing a third quality control process on the first ground wave radar data to obtain third quality control data; When the data type belongs to the fourth type, performing a fourth quality control process on the first ground wave radar data to obtain fourth quality control data; Obtaining the second ground wave radar data according to the first quality control data, the second quality control data, the third quality control data and the fourth quality control data; Among them, the first quality control process is to identify outliers by standard deviation, the second quality control process is to identify outliers by box plot, the third quality control process is multi-dimensional outlier detection, and the fourth quality control process is to identify outliers by machine learning.

3. The automatic processing method of ground wave radar according to claim 2, characterized in that: When the data type belongs to the first type, performing a first quality control process on the first ground wave radar data to obtain first quality control data includes the following steps: According to the spatial data set of the first ground wave radar data, obtaining a spatial sample capacity, first observation data, and a first average value of the first observation data; Obtaining a first standard deviation of the first observation data according to the spatial sample capacity, the first observation data, and the first average value; Obtaining a first absolute value of a difference between the first observation data and the first average value; The first observation data corresponding to the first absolute value greater than twice the first standard deviation are eliminated to obtain first intermediate data.

4. The automatic processing method of ground wave radar according to claim 2, characterized in that: When the data type belongs to the first type, performing a first quality control process on the first ground wave radar data to obtain first quality control data further includes the following steps: According to the time series data set of the first ground wave radar data, obtaining a first time series sample capacity, second observation data, and a second average value of the second observation data; Obtaining a second standard deviation of the second observation data according to the first time series sample size, the second observation data, and the second average value; Obtaining a second absolute value of a difference between the second observation data and the second average value; The second observation data corresponding to the second absolute value greater than twice the second standard deviation are eliminated to obtain second intermediate data.

5. The automatic processing method of ground wave radar according to claim 2, characterized in that: When the data type belongs to the first type, performing a first quality control process on the first ground wave radar data to obtain first quality control data further includes the following steps: According to the time series data set of the first ground wave radar data, obtaining a second time series sample capacity, third observation data, and a third average value of the third observation data; Obtaining a third standard deviation of the third observation data according to the second time series sample size, the third observation data and the third average value; Obtaining a third absolute value of a difference between the third observation data at a current moment and the third observation data at an adjacent moment; The third observed data corresponding to the third absolute value greater than twice the third standard deviation are eliminated to obtain third intermediate data.

6. The automatic processing method of ground wave radar according to claim 2, characterized in that: When the data type belongs to the second type, performing a second quality control process on the first ground wave radar data to obtain second quality control data comprises the following steps: Obtaining a lower quartile and an upper quartile of the first ground wave radar data; Obtaining the difference between the upper quartile and the lower quartile to obtain the interquartile range; Setting a lower edge according to the lower quartile and the interquartile range; Setting an upper edge according to the upper quartile and the interquartile range; The first ground wave radar data smaller than the lower edge and the first ground wave radar data larger than the upper edge are eliminated to obtain the second quality control data.

7. The automatic processing method of ground wave radar according to claim 1, characterized in that: When the data type belongs to the third type, performing a third quality control process on the first ground wave radar data to obtain third quality control data comprises the following steps: Extracting features from the first ground wave radar data to obtain multi-dimensional features; Performing data cleaning on the multi-dimensional features; Set the outlier ratio threshold; Constructing an outlier model based on the multi-dimensional features after data cleaning; According to the outlier ratio threshold and the outlier model, outlier detection is performed on the multi-dimensional feature to obtain a first detection result; According to the first detection result, outliers are eliminated from the first ground wave radar data to obtain the third quality control data.

8. The automatic processing method of ground wave radar according to claim 1, characterized in that: When the data type belongs to the fourth type, performing a fourth quality control process on the first ground wave radar data to obtain fourth quality control data comprises the following steps: Initializing an isolation forest model and training the isolation forest model; According to the trained isolation forest model, performing outlier detection on the first ground wave radar data to obtain a second detection result; According to the second detection result, outliers are eliminated from the first ground wave radar data to obtain the fourth quality control data.

9. The automatic processing method of ground wave radar according to claim 1, characterized in that: When the original data belongs to multiple ground wave radars, performing multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data includes the following steps: converting the second ground wave radar data belonging to a plurality of ground wave radars into a common grid; In the common grid, second ground wave radar data of non-overlapping areas in the observation areas of the plurality of ground wave radars are saved as fourth observation data; In the common grid, performing secondary vector synthesis on the second ground wave radar data of the overlapping area in the observation area to obtain fifth observation data; The fourth observation data and the fifth observation data are combined to obtain the third ground wave radar data.

10. A ground wave radar automatic processing system, characterized in that: include: The first module is used to obtain the receiving antenna pattern, calibrate the channel amplitude and phase according to the receiving antenna pattern, and obtain the original data of the ground wave radar observation data; The second module is used to decode the original data to obtain first ground wave radar data; A third module is used to perform quality control on the first ground wave radar data to obtain second ground wave radar data; A fourth module is used for performing multi-station data fusion on the second ground wave radar data to obtain third ground wave radar data when the original data belongs to multiple ground wave radars; The fifth module is used to assimilate the surface flow field of the ground wave radar according to at least one of the second ground wave radar data or the third ground wave radar data through a data assimilation module to obtain target data of the ground wave radar observation data.

Images